My iPhone 4S doesn't connect to headphones?

I got my iPhone 4S brand new very recently, but when I plug my headphones into it, it continues to use the speaker's sounds (the speaker has been working completely normally). I tried two different headphones, the ones that came with it and my own. I am sure that I plugged them in all the way. Is there a hardware problem, or am I doing something wrong? What can I do to make the headphones work?