Disconnected Homepod from Apple TV and now am unable to have audio coming through TV speakers

So about a week ago, I went to watch tv before bed and realized that the audio was now coming from my homepod. The night before, it was coming through the tv so I thought that was weird but didn't think too much of it. I decided to go into setting and change the audio output to my tv speakers, but to my surprise, no audio coming through. I tested my audio on my TV and it works perfect, It's just whenever I am on the Apple TV that the audio is nonexistent unless its coming through the homepod. My default audio device is "TV Speakers" and so is my temporary output device. I have tried it with no temp device and one with one and neither work. Can anybody tell me if I am doing something wrong or if there's something I am missing?