As with most questions, the answer is "depends." With grains, it depends on which grains. If you are adding a small amount of dark roasted malt for color, an ounce or two either way can make a big difference. An ounce or two of your base malt, not so much.
For yeast, you don't want to come up short. That will leave you, potentially, with under-attenuated beer and undesirable off-flavors. Better to err on the side of more rather than less.
With salts and minerals, if you are going to mess with them at all, I'd suggest that precision is the key. (My personal opinion is that the addition of these is rather over-rated.)
Speaking more philosophically, I think the answer really depends on you. Are you looking for consistency, or do you enter beers into competitions? If so, you really want to be precise in your measurements so that you can replicate prior successes, or make sure your beer really fits the right category. If, like me, you are brewing for a market of one, and consistently make really good beer, you can be a bit more lax.
However, most of us have, or should have, a goal of becoming a better brewer. Being careful with your measurements then becomes somewhat more important, because you want to be able to replicate your prior successes, and avoid prior pitfalls.
When I design a beer, I have a particular goal in mind -- usually a flavor profile, though sometimes I'm merely experimenting with a new ingredient, or want to hit a precise color. By measuring carefully, I'm more likely to hit the mark at which I am aiming, or am better able to taste the impact a new ingredient made. If you just throw stuff in willy-nilly (not that you are doing that), you run the risk of making beers that are wildly inconsistent in their quality. One beer will be good, the next one not.
That said, you do not need to measure most of your ingredients down to the milligram.