Homepods as Apple TV speakers

I've had two full-sized homepods in my upstairs office for some time. I recently bought two Homepod minis, thinking I could use them downstairs as the speakers for my Apple TV. When setting them up, I read that I needed the full-sized ones to do that. So I swapped them, putting the minis upstairs and the big ones downstairs. The Apple TV can't seem to connect to the Homepods even when I set them as the default speakers. Sometimes I can get them to play music in stereo, but at other times they won't even connect to Apple Music. The minis are working fine. Any ideas?