Short answer is yes, but you will lose some carbonation. So try and compensate for that.
But if it will all go into growlers, why wouldn't you condition in the growlers?
If you intend to force carb the keg, then you've got the right idea, but i'm reading that you are conditioning in the keg, not force carbing. Is that correct?