Headphone jack problems.

I bought the new iphone 4s Saturday and I just tried using my headphones with it. I tried my Bose headphones first and they worked for a little bit and then the music started playing through the speaker. I tried them again with the Bose headphones and it didn't play at all even if I held them in. I tried the headphones that came with my iphone and they only work if I hold them in. Weird. I haven't damaged anything on it, do I don't understand why it's not working. Any help?