Why does only one ear work on my headphones on my 5s it's a new phone

I Just Got The iphone 5s and from the day I got it one ear on the headphone wasn't working and the same thing was happening to my old iPhone 4S is it just my phone or is this every phone it's only when I'm listening to music when I use it during a phone convo or FaceTime or plug somewhere else it's perfectly fine