No airpod audio on incoming or outgoing calls unless...

I have reset and re-paired my 1st gen airpods on my iPhone X. When I take the airpods out of the case they sync with the phone and I hear the paired sound effect.

Then, if I make a phone call or receive a phone call, they're dead silent and the mic doesn't work. The phone shows connected to the airpods at this time.

To fix it, I have to briefly enable speakerphone and then switch back to airpods in the audio selection menu, and it works fine for that call. The problem starts all over again on the next call.

I have everything up-to-date, charged, reset and I seems like there is no fix for me.