Right after you changed the settings to rotate the screen instead of mute, did you double tap the home buttonand look in the task bar then? Just because you changed it in the settings doesn't mean that the screen could not have been locked in the task bar. You may still have to unlock it in there and the setting change takes place.
System sounds can be muted and controlled two different ways. The screen lock rotation can be controlled in the same manner as well.
Settings>General>Use Side Switch to: Mute System sounds. If this option is selected, the switch on the side of the iPad above the volume rocker will mute system sounds.
If you choose Lock Screen Rotation, then the switch locks the screen. If the screen is locked, you will see a lock icon in the upper right corner next to the battery indicator gauge.
If you have the side switch set to lock screen rotation then the system sound control is in the task bar. Double tap the home button and in the task bar at the bottom, swipe all the way to the right. The speaker icon is all the way to the left. Tap on it and system sounds will return.
If you have the side switch set to mute system sounds, then the screen lock rotation can be accessed via the task bar in the same manner as described above.
This support article from Apple explains how the side switch works.
http://support.apple.com/kb/HT4085
