It's a matter of personal preference. I dump everything but the hops into my primary and get very clear beer and the sludge does not have an effect on flavor.
Here are my observations on flavor, evolving from extract to mini-mash to AG over the years:
I realize that this statement is blasphemous to some, but I have never gotten what I would consider robust malt flavors from extract/kits. Even a mini-mash, in my hands, is a thousand times better. I also find that the choice of base malt has a huge impact on the flavor of the finished beer. The whole reason I ended up going AG is because my beers were, exactly as you describe, "lacking some flavor."
A clean fermentation (with the right yeast) and proper attenuation makes a huge difference, which is something that I underestimated in the beginning. For me that means making starters or using good quality, re-hydrated dried yeast and aerating properly.
To get hop flavors to play nice with the grains, you need to carbonate properly. You also need to adjust the level of carbonation to the recipe/style. It doesn't have to be exact, but there is a big difference in flavor between 1.5 and 3.0 volumes.
If you're trying to get a crisp, hop nose, try adding a truckload of hops right after switching off the burner (and cooling the wort rapidly). I have found that even 5 minute additions can step all over the body of the beer and dry-hopping just doesn't give the same crispness.
Don't underestimate the value of quality time in the secondary. A solid few weeks in the secondary allows time for the flavors to blend and for the subtleties of the grains to come out. (A couple of months in a bottle will accomplish same.) Of course, the time varies greatly by style.