How to STOP AirPods from asking to connect to the Apple TV

Have an AppleTV 4K (previous model, re ATV4K) with a stereo set HomePod 2s, and AirPods Pro 2. Since I'm older now, I'm using at least one of the AirPods as a hearing aid to better hear what's playing on the ATV4K (and to avoid playing it too loudly, thus making my wife as deaf an I am). Unfortunately, the ATV4K keeps flashing up a screen prompt and continually tries to connect to my hearing-AirPod (thereby ditching my speakers). Often trying to cancel the connection prompt switches the audio anyway (in the dark, it feels the same) and I end up cursing the remote.

Is there any way to disable this automatic connection? It should be something we can disable since sharing the video and audio with the HP2s is the primary function, and constantly requesting AirPods is irritating & distracting.