Like most, I usually brew alone. Occasionally I'll invite people over and make a big event of it. Alot of people have mentioned screwing something up with people there. Anytime I brew with others, I always make sure I'm prepared in advance. When alone, I'll do things like mill grain, measure hops, etc... during the downtime. When I brew with others, I do all of this in advance. I show up to brewday with a bucket of milled grain, ziploc bags of hops that just say "60", "15", etc... If the event is at my house, I get the burner, pots, etc... ready and in place the night before (in the garage). I also measure my water in advance and set my electric HLT to the proper temperature. Also, as soon as the boil starts, I set reminders in my phone at each hop drop and flameout.
When I do all this, the downtime to hang out really is downtime. It's SOO much more relaxing that way.