iPhone X Call Audio Routing Automatic setting confusing

I have an iPhone X. When receiving calls, I like them to go to Speaker mode, so I have Call Audio Routing set to "Speaker". Ex. when calling any company, I usually have to go through a menu ("Press 1 to do this, Press 2 to do that"), so it's more convenient to do this in Speaker mode. However, there are times when I like to use my AirPod Pros. If I set Call Audio Routing to "Automatic", the calls will route to my AirPods. However, if I forget to switch Call Audio Routing back to Speaker, the next time I make a call, it will default to Phone mode. I'll push the Speaker button, but as soon as the other person answers, it switches back to Phone mode.

Is there a way to make "Automatic" default to Speaker mode? Or do I need to keep going back to Call Audio Routing to get the customized experience I'm looking for?