Audio output device keeps changing

I have Phonak hearing aids which are not MFI (please do not refer to any solutions for MFI hearing aids, they do not apply to me), but rather show as Bluetooth devices. I can stream audio or notifications to them, but I rarely want to. I mainly want to use them for phone calls, and that works great.

The problem is that the phone keeps switching spontaneously to the hearing aids as the main audio device, and suddenly I get a loud notification tone in my ear. How can I configure it to only use the hearing aids as the audio device when I select it?

Here are some screenshots. The first is the way I want it to stay. I can make and receive calls with the aids, but all other audio goes to the phone. The second screenshot is what I do NOT want.