Why doesn't the mod wheel work with Garageband 3 organ sounds.

The mod wheel only works on certain GB instruments - basically those which could do vibrato such as the classical guitar or the Hollywood Strings. It's not available with instruments which can't do vibrato, and Apple obviously consider this applies to the organs. Strangely, I was able to produce vibrato on the Grand Piano!

You might be able to find an organ SoundFont which would accept vibrato.

Try using your sustain pedal for changing the Leslie speed. It is VERY realistic. Sounds just like a Leslie switched between slow and fast rotation. Add an amp simulation setting like "Big Wheels" with the Gospel or Rock organ. It sounds like you are distorting the Leslie amp. Real fun!