Check the Music app settings on your Apple devices.
For iPhone, you can stop Autoplay by going to the Music app, tab the mini-player at the bottom, finding the ∞ infinity symbol in the mini-player. The list button in the lower-right of the player window will bring up the ‘Playing Next’ queue list. It will have three symbols on the right side of the words ‘Playing Next’: Shuffle (🔀), Repeat (🔂), and Autoplay (∞). If shown inverted, tap it to disable. Once disabled, the music player shouldn’t play beyond the current track, and (if I understood it correctly) shouldn't seek to connect to discovered AirPlay and Bluetooth speakers automatically and play there.
It seems to be about mobile devices and the HomePod speakers, rather than Apple TV?
Check the HomePod forum for more expertise on that device.