Can't control Homepod Stereo Pair Volume with Apple TV 4K remote

So I have my Homepod minis connected as a stereo pair, in the same room as my AppleTV 4K, and I want the sound of my AppleTV 4K to come out of the stereo pair. This works when I use AirPlay, and when I first set it up, I could control the volume with my Apple TV's remote, but now I can't control the volume of the stereo pair using the Volume controls. The only way I can get it to work is by holding down the TV button, and choosing "Airplay" then selecting the "Homepod Pair" and adjusting the volume. I can no longer just change the volume while simultaneously watching a show on the Apple TV 4K. How can I get this to work? - thnx