I had the same issue as you. I started with my 40 GB iPod to use at the gym, then it became a Mini, then a Shuffle.
It's very small and light to take to the gym (Nano). It's the same length as a credit card but cut about 1/2 inch to its width.
This Nano was intended to replace my Mini which is in my car, but it's definitely gym material. However, I wouldn't take it to the gym until they made a case for it. I don't want excessive scratches and it might be vulnerable to sweat as well.
When it comes to the gym, bigger is not always better, and I do enjoy using my shuffle. I never needed a screen while working out because if I don't like the song, I just skip to the next one. The only reason why you would get a Nano to work out are: you prefer a screen (which is beautiful, BTW) and the functions of an iPod Photo. If you're on a budget and want this model, consider the 2 GB version. 500 songs in the gym would last your for days. I only average 1-2 hours in the gym daily and my playlist of 200 is more than enough to satisfy my needs.