Is that a 4th gen shuffle, or maybe a "6th gen" nano? The 4th gen shuffle is the current shuffle.
If it's a 4th gen shuffle, and you are using the USB cable that came with the 4th gen shuffle on a 2nd gen shuffle, it will not work. The connector is different between the 2nd gen shuffle and the 3rd/4th gen shuffle.
This is the replacement part for the 3rd/4th gen shuffle
http://store.apple.com/us/product/MC003ZM/A
It says, toward the bottom, that it only works with the 3rd and 4th gen shuffle.
If you have the USB cable (with a mini-dock at the iPod end) that came with the 2nd gen shuffle, you need to use it instead. This is the replacement part
http://store.apple.com/us/product/MA694G/A
You can get a less expensive third-party replacement (most just have a connector instead of a dock) on Amazon or eBay. Just make sure it is described as being for the 2nd gen shuffle.
If that other iPod happens to be a 6th gen nano (or other non-shuffle), please post back with clarification.