Do you get sound in any apps e.g. Music and Videos ? If you do then have you got notifications muted ? Only notifications (including games) get muted, so the Music and Videos apps, and headphones, still get sound.
Depending on what you've got Settings > General > Use Side Switch To set to (mute or rotation lock), then you can mute notifications by the switch on the right hand side of the iPad above the volume switch, or via the taskbar : double-click the home button; slide from the left; and it's the icon far left; press home again to exit the taskbar. The function that isn't on the side switch is set via the taskbar instead : http://support.apple.com/kb/HT4085