Weird audio problem during calls

I have a weird audio problem and was wondering if anyone else has had a similar problem or could help...

My wife and I both have iPhone 5's and sometimes (not all the time) when she calls me (or I call her), it connects, but no matter which audio input I choose (phone, bluetooth, speaker, headphones), I cannot hear her, but she can hear me... I hang up, call back (and when I call, there is no dialing sound, nor any other phone sounds) and it's the same thing. Sometimes restarting the phone works, other times I have to restart it 3-4 times before it'll work again. I try different things during these calls, like turning on/off bluetooth and/or wifi (pretty much anything I can think of that would make a difference), but nothing seems to help.

The (other) weird part is that after the last time of this happening, I tried calling my iPhone with my office phone and it worked fine, so then I immediately call my wife back with my iPhone and it was doing the same thing again. I called her back using my office phone and could talk to her fine...

This problem has been going on for a few months now and I have tried restoring the phone a couple times (and I am on the latest iOS) and sadly, that hasn't done anything to help... Does anyone have any ideas what is going on? I'd make an appointment for the problem, but since it doesn't do it all the time, I don't know what good it would do... But it is getting very frustrating!

Thanks for any help!

-Jason