Have your earbuds scratched your ear?

I have noticed that my Iphone 5 earbuds have been scratching the inside of my ears. The port on the side is rough if you run your finger across it, and your ear is pretty sensitive. This is particularly an issue if I lay down while listening to music. Anyone else experience this?