On the 3rd and 4th gen shuffle (unlike earlier shuffles), it plays songs from the All Songs list unless you use VoiceOver to switch to a specific playlist that you loaded on the shuffle. From the All Songs list, songs are obviously shuffled during playback if the power switch is set to shuffle. If the switch is set to play-in-order, songs are played in alphabetical order. It may not be obvious that the song playback order is alphabetical; it may seem shuffled.
To play songs in playlist order, the songs need to be on a playlist that is loaded on the shuffle. Then, when using the shuffle, use VoiceOver button to select that playlist. This document has information about setting up VoiceOver on the 3rd gen shuffle
This document has additional information about using VoiceOver; although it's for the 4th gen shuffle, I believe it works similarly on the 3rd gen shuffle