It's not the color, it's just that darker beers tend to be larger gravity beers, and THEY take longer. You can make a black or amber lager and it'll take the same amount of time as a fizzy yellow lager.
Maybe it's because I'm comparing a Belgian Wit, with around 4-4.8% alcohol, versus stouts and dunkels that are probably closer to 5.6-6%. So it's really just the alcohol level?