AirPods switch to iPhone audio mid conversation

With my AirPod pro in my ears while on a phone call, with my phone in my pocket, the audio will randomly switch back to the iPhone. The AirPods aren’t disconnected because when I open the phone and click on the audio button, the AirPods are still listed. A similar thing when I am listening to music or podcast. It randomly pauses and I have to pull my phone out of my pocket and hit play again. This is quite frustrating as I have ear buds on for a reason. I need to be using my hands for something else, so constantly have to have to unpause or fix the audio is a problem.