SOLUTION!

I have found the exact same behaviors and have wasted many hours and gone down countless roads to try and figure this out. The only way to make the issue go away is to ensure your Apple TV is added to HomeKit. Problem gone! Why Apple would engineer this or why they would not know how to resolve the issue is inexcusable.