Using my iPhone as an Apple TV remote

I have successfully paired my iPhone with my Apple TV for the second time. It used to work perfectly, but since I got a new Apple Watch, it has stopped working as a remote. What’s interesting is my Apple TV remote app on my new watch actually works. What’s crazy is screen mirroring works until I try to mirror a video or play an Apple fitness + video on the tv. Then it doesn’t work again. I have everything updated to most current software, restarted all of my devices, and I have tried doing a hard reset of my iPhone. However, once I use my iPhone backup and restore the apps and all, my Apple TV remote (from the drop down list in command center) does not work. Is there any solution for this?