How to disable HDMI audio output on Macbook Air (2019, Catalina)

I have a recurring problem with the audio and I cannot find an answer. I use the external headphones for audio output. But when I plug in an external monitor (HDMI to adapter to USB-C) the audio sometimes switches to that output.

Even when I use the audio settings to return the audio output to external headphones, it will be unstable, switching back and forth between the two chaotically. When this happens I can't listen to anything for more than a few seconds before the Mac stops recognizing the external output and audio stops.

I even downloaded an app for this (Audio Profile Manager) but it doesn't consistently fix the problem. It seems like this issue goes back years, so it's surprising to me Apple has not fixed it. Any suggestions?