Airpods pause automatically when you come near the iPhone (listening to music with Apple Watch in the gym)

I work out exclusively with my Apple Watch while exercising and listen to music through it.

I keep my iPhone in a locker.

That works so far, but whenever I get close to the iPhone, the music pauses and I have to restart the playback myself.

This is especially annoying when you are exactly at a distance from the iPhone where it keeps disconnecting and reconnecting. The music pauses every time.

Is there a way to have the AirPods stay connected to the Apple Watch only and ignore the iPhone (other than turning off Bluetooth)?

Surely that can't be a desired behavior?

Thanks in advance.

~Maik