Change default Audio app for Airpods?

How do I tell my phone and/or Airpods WHICH audio application to start up when I double-tap the airpod?

I have the airpods set to "Play/Pause" when I double tap but there is no way to tell the device WHICH application to Play/Pause.

It seems by default that the phone always wants to start Apple Music, which is an app I never use. 95% of the audio I listen to is podcasts played through Apples Podcast app. The remaining 5% is Audiobooks through the Audible app.

If I have been listening to a Podcast in the last hour or two and the app is still running in the background, then double tapping will usually restart the podcast. But I find that if the Podcast app is no longer in the background and I haven't played a podcast in the last couple of hours, it's a complete crapshoot as to which app the phone wil start up, and the preference with the phone seems to be the Music app. This is in spite of the fact that I NEVER start the app on my own or listen to any music through it. The only time the Apple Music App has ever been started is by the flawed AirPods functionality.

Even worse, as I don't use the Music App and this activity isn't desirable, I have deleted the Apple Music app from the phone altogether. Without the Music app installed the device should just default to the most commonly used app. Instead what happens is that, if the PodCast app isn't running in the background, a double-tap simply does nothing. I have to take the phone out and manually start up the Podcast app.

What's the point of having wireless headphones that control a device with multiple types of multimedia apps (music, audiobooks, podcasts, telephone etc) if I don't have the ability to tell the device which application it should be controlling? Being able to assign Play/Pause functionality is pretty worthless if I can't decide which application is to be played!!