Airplay volume auto defaulting down/low when changing content

Hoping I can find a solution to this, curious if folks are running into the same issue.

For AppleTV 4k (newest model) audio I use the Airplay output to my hifi system (Holo Red Streamer with Gentooplayer software which actually runs Airplay 1 for better quality at the expense of a slight delay in audio loading and syncing). Then I digitally control volume on the AppleTV via the included remote. This setup has been fabulous and I have not missed using HDMI eARC at all. Until recently at least.

I am having an issue that when I first turn on the Apple TV, or switch apps/streams, etc the AppleTV automatically turns down the volume from where I previously had it set at. Like it thinks it needs to keep resetting to some sort of default spot whenever I am switching between anything. Not sure why this started but it is a huge pain as it turns it down low enough that I am hardly getting any output (family sometimes thinks the airplay connection is broken or thinks they need to go crank my amp), and I have to turn it back up constantly while using the AppleTV. The analog volume I keep my Hifi system at means usually using the AppleTV a handful of clicks from the top of the volume slider, and it lowers it to like 20-30% volume everytime I change what content I am streaming. It is maddening.

I am new to the Apple ecosystem, AppleTV came first and I recently switched to iPhone after liking the AppleTV so much. I cannot for the life of me figure out what is causing this to happen though. Anyone have any fix suggestions?