How to stop Apple TV remote on iPhone from streaming everything to TV?

i have an issue where every time i control my apple tv VIA software remote on the iphone it makes everything i do stream to the tv. im not casting, im not screen sharing. example... cant find apple tv remote, open software remote to control tv then… then lets say i open Apple Music and want to play a song, it will stop what’s on the tv and immediately play to the tv. At the bottom of Apple Music it shows it’s being casted to tv without ever selecting that option and unchecking Apple TV to stream to phone doesn’t always show as an option to switch to. It’s automatically being handed off to the tv and it’s annoying.

[Re-Titled by Moderator]