How do I get my iPhone to play sound thru earbuds only?

I wanted to listen to some music on my iPhone 4 last night without disturbing my wife, but I can't seem to find a way to shut the speaker off.

Sound plays through the earbuds, but also through the speaker.

Is there any way to get the phone to play sound through the earphone output only, while the earbuds are plugged in?

That little sliding switch on the side of the phone doesn't seem to have any effect.