iPhone call audio on speaker even when wearing AirPods

Hello! I have my iPhone 13 Pro Max set to answer all calls on speaker, which I love (Settings->Accessibility->Touch->Call Audio Routing->Speaker). But, when I have my AirPods in and I am listening to music, if I get a call, my iPhone answers the call on speaker. This is very inconvenient when my phone is in my backpack while hiking or in my armband while running (I have to take out my phone to speak into the speaker).

What I would like is for my iPhone to answer all calls on speaker EXCEPT when I am wearing my AirPods, then I would like to call to come through my AirPods, NOT my speaker.

I have tried Settings->Accessibility->Touch->Call Audio Routing->Bluetooth Headset but this means that when I do not have my AirPods in, the call audio defaults to iPhone, not speaker.

Does anyone out there know how to configure this specific behavior?