Apple tv + homepod default speaker issue

I have now Made my homepods the default speaker for my apple tv and it works great. However I can no longer use my homepods as standalone speakers, for example if I want to watch youtube on my ipad but stream the audio to my homepods, it will now turn on my apple tv and start playing the video there, same with music. Is there no way around this? As I love the fact that my homepods are now my default speakers for my TV, but still want them to act as standalone homepods when my apple tv is off.