No Sound On iPhone When Called

I have an iPhone 5S and my wife has an iPhone 5 - both are on iOS7. This problem is on both phones.

When we receive a call, we can't hear the caller but the caller can hear us fine - regardless of whether we have phone to our ear or on speakerphone. We usually have to tell them to call back. On the second or third attempt, we can hear them fine.

This doesn't happen every time but probably 25-35% of the time now and getting quite frustrating.

Any thoughts on what's wrong on both phones?