Now that the Nano has come out though its a tough one. Which would you prefer and why?
Let me toss a question back at you: what kind of exersize? For example, the Nano comes with a "lap" timer, so if you like to check your speed on runs or laps on a bike, the Nano's going to be more useful to you there. Likewise if you like to change playlists a lot to match what you're doing--one for weight lifting, one for the stationary bike, etc. The Nano will make it easier for you to match playlist to exersize. Not that you can't do it with a shuffle--but you have to arrange things ahead of time, put the weight-lifting playlist on the shuffle--you can't just, on a whim, say, "Naw, changed my mind, don't want that playlist, I want this other one for for weightlifting today...."
If, however, like me, you usually listen to just one playlist (or shuffled music) while working out...well, the shuffle will do you just fine. For that, you don't need a screen, don't need more music choices, don't need any extras at all. The shuffle is very minimalist and is really an excellent choice if your needs are equally modest.
That said, I'd go for the Nano; yes, it's overkill for the gym, but I'm betting that once you have one, you'll use it elsewhere. I think, for example, that it's perfect for waiting in line and airline travel--especially with the headphone/lanyard so you can hang it about your neck, leaving your hands free to haul and manage those bags.
Remember that it can hold photos, has a clock, calendar, and contacts list. So you can use it to check the time, appointments, find phone numbers, etc. All the shuffle does is play music. That's certainly enough for an hour at the gym two or three times a week--and if that's ALL you're going to use it for, save your money. But is that all you're going to use it for? That's the $100 dollar question (as that's how much more the Nano's going to cost you).
If you do go for the Nano, keep in mind that you may not be able to get certain accessories right away--like the armband or lanyard. It may take up to 4 weeks for all such accessories to be avalible. So you'll need to slip the Nano into some sort of pocket or temporary holder if you get it right away and start taking it to the gym.