Apple TV on Xbox Series X has HDR issue

Why is it that when I bring up the Apple TV app on my Xbox Series X is menu is in HDR, and when I start a show it is in HDR for a second, but then switches my TV back to SDR mode and washes out the picture? But I think the signal is really still HDR because if I force my TV to HDR mode from auto the picture looks fine. But I want to be able to leave it on auto as switching back and forth constantly isn't realistic. My TV doesn't support Dolby Vision, so is it maybe trying to change to that which isn't available so it defaults to SDR?