I don't use a secondary fermenter. As well, I noticed that my beer tends to take on a bit of a yeasty taste to it. Would a secondary fermenter prevent this yeasty taste?
My first question might be obvious but I gotta ask it: Are you pouring the yeast sediment from the bottle into your glass? That would certainly do it.
Assuming you're not, can you be more specific about the flavor? Is it fruity? Medicinal? Green apple?
What yeast strain are you using and what temperature are you fermenting at? Most problems with off-flavors from the yeast come from fermenting too high.
Besides the afore mentoned reasons, often it tastes too yeastie because it is being bottled and served too young. You don't need to secondary, you can do what many of us opt for and leave it in primary for a month.