Most of the time I'll rock or swirl mine gently to wash that off the sides. If you have left it on there too long, you might not get as much off as you'd like.
Does it hurt to swirl the beer and put the stuff back in the beer? Does it hurt if you don't get that stuff back in the beer?
I don't know. I've had good enough beers doing either.