iphone - can't hear caller for first 5-10 seconds of call !!

I have an phone 5S. I am up to date with iOS 11.4.1. I use AT&T.

Aprox 5 calls out of 10, when I answer a call in my home, I can not hear the caller for the first 5-10 seconds. The caller can always hear me fine. I usually say something like "Hello ?! Hello ?! Hello !!", since I can not hear anything. Eventually when I can hear, the caller usually says something like "What's wrong with you ?? Why did you keep saying Hello ??!!" This is really frustrating for me and the callers !! I have tried Powering off/on; Airplane Mode on/off; Mobile Data off/on. Sometimes the problem clears up, but then comes back a few days later. In addition, calls occasionally get dropped. I don't know if this is related. (FaceTime and WhatsApp calls are much better and clearer than using traditional AT&T phone connection.)

Any ideas how to rectify this please ?? Thanks !!