Apple TV with Bluetooth audio: How to get fixed volume?

I'm using my Apple TV with HDMI to my projector with no speakers. I have an external bluetooth-receiver connected to my non-intelligent amp+speakers. I have a nice stereo setup and I am happy enough to wathc/listen to Apple TV in stereo over the bluetooth. No need for fancy surround system. For my music needs, I got a completely different wired setup with the same amp/speakers. All fine and dandy up until now.

My problem is not being able to control the amp via Apple TV remotes IR, whilst having a bluetooth connection out of Apple TV.

I would like to have constant fixed BT-audio out of Apple TV and ability to control the amp volume with the Apple TV remote IR. When I use BT-audio, Apple TV remote just wants to control the BT signal going out and just drops the IR ability.

I do understand that controlling some BT-receivers, like my headphones, it is handy to have the BT-volume control in the remote. Works a charm with my Airpods Max. I just would like to have the ability to change my BT-receiver settings so that I could have BT-audio out (for some of the devices) with fixed volume and at the same time have the ability to control my amp with IR.

Thanks in advance, if anyone knows a fix. :)