how do I delete music from my Shuffle?

Which shuffle model do you have? This document shows the four types

http://support.apple.com/kb/HT1719

1st and 2nd gen works a bit differently from 3rd and 4th gen.

Also, if it's 3rd or 4th gen, do your load the shuffle using automatic syncing, or the manual method (drag and drop songs on the shuffle in iTunes)?