More than likely you will need 12v to make it spin.
I noticed on the one I was building I was using a power supply that was not quite 12v and it would bairly spin and constantly stop. If I pushed it with my finger it would spin for a second then stop again. When I put a little extra weight on it with a Hard Drive magnet that it would not spin at all.
Once I move to a 12v power supply and shielded the magnet from the fan motor it worked just fine.
Also make sure the box you are using is thin enough the magnet works well through. I used a plastic project box from Radio Shack.