I notice that cleaning is what tires me out. I brew 11 gallon batches, but I have pumps and a tippy dump so it's not too much work. I sometimes clean in place, but when I take it apart to clean every once in a while, I feel beat afterwards.
I use kegs for my HLT, MLT, and BK. I never move the HLT or that pump (as it's always been water only), but hoisting and cleaning the MLT and BK is a pain if I"m tearing it all down. I think that a keg is about 30 pounds (?) empty. They are a bit harder to manipulate into my brewery sink with the connections, the hardware for the connections, the electrical stuff, etc and somedays it's tougher than others.
I weigh about 130 pounds and I'm 52 years old. By the end of a brew day, I'll have lifted and measured 25+ pounds of grain, and then hauled it up the stairs from the basement just to start, mashed in with it, dumped it (wet it weighs lots more!), and then cleaning adds tons of weight. By the end of the day, I'm sure I've lifted more than twice my own weight. I'm in good shape but it's not very much fun anymore.
I always think when I do smaller batches that I'd get a picobrew or grainfather or something.