Headphone Adapter Issue

So I have a Lightning to 3.5 mm/Lightning Adapter, and i'm able to hear clearly with them. The issue is that whenever anyone calls me, I can't hear the phone call through the headphones, but once the call is over, I can hear everything again. I always have to either hear from the phone itself to talk to the person, or put the phone on speaker to hear them. Can someone please tell me why the headphones won't work when making a phone call, but works with everything else?