You should always get it off the bulk of the lees early after primary as this can cause unwanted flavors and offers no benefit. Small amounts of sediment settling out over time aren't as big of a concern.
With multiple rackings you need to be more concerned with oxidizing. If your not racking...