Airpods Max automatic head detection doesn't work for phone calls

When not wearing my Airpods Max, automatic head detection works fine for music, etc. However, every phone call routes through the headphones whether I'm wearing them or not. I've tried turning detection off and on again, resetting them, etc., but the problem persists. If I don't put them away in their case every time I take them off my head, for every phone call I either have to run through the apartment scrambling to find them, or manually select the audio option for the call to set it to phone, which takes a few seconds to kick in after the call has started.

Any ideas for how to make sure auto head detection also applies to incoming/outgoing phone calls?