Issues playing music or podcasts when using HomePods as AppleTV default speaker

I've been around and around on this--even spoken to support on phone, sent in reports, all to no avail.

When I set up two HomePod minis (set up as stereo pair), they work fine playing music, podcasts, etc. Then when I set up AppleTV with HomePods as default speaker, the music playing, etc works for a while, but soon after it will no longer respond to requests to play music, podcasts, etc--either from voice request with Siri or with Handoff from iPhone. Issues like "Sorry, X, there seems to be a problem with Apple Music" or "cannot connect" messages come up.

I've done about everything I can to fix...unpaired, set up just one, reset, unplugged, new wifi set up, you name it, I've tried it. I THINK. What am I missing? Does anyone else have this problem?

(Saw an old thread about this same issue from 2020 but it was closed, so I started this new one).