Receiving Calls - Caller can hear me but I can't hear them

When I receive calls 50% of the time the callers can hear me but I can't hear them. When I tell the person on the other end, "Sorry, I can't hear you, please call me back." and they do, it always works and things are fine. It's only on the initial call in. This is of course quite frustrating. Since the second call always work, it points more to a software issue rather than hardware. I've tried restarting the phone as well as restoring it to no avail. I'm running the latest iOS and the phone is fairly new as I had it replaced not too long ago for a battery problem I had (and many others...don't get me started on that!).

Anyone experienced this and have a solution?