Q: 6 plus - Hard to hear caller

I really love my 6 plus, however, I've noticed on many occasions that my I have a difficult time hearing the other caller unless I place the call on speaker.  It's almost muffled.  I can't seem to find a setting to adjust this or is this a phone defect?  I know I'm not going deaf.