You still have it turned 'on' in Settings > Sounds ? If you do then have you got notifications muted ? Only notifications (including games) get muted, so the Music and Videos apps, and headphones, still get sound.
Depending on what you've got Settings > General > Use Side Switch To set to (mute or rotation lock), then you can mute notifications by the switch on the right hand side of the iPad above the volume switch, or via control centre : swipe up from the bottom edge of the screen and it's the right-most of the 5 icons in the middle of the control centre. The function that isn't on the side switch is set via the taskbar instead : http://support.apple.com/kb/HT4085