Latest Apple TV 4K, Delay when going back

I suspect that you're experiencing your TV switching between various UHD/4K HDR video modes as the AppleTV device switches from one mode to another (from a movie back to the home screen for example).

I've noticed the same thing, and it feels like your device is sluggishly hanging up for a few seconds at a time. Frustrating, since we know the Apple TV 4K has a decent chip in it and it's usually pretty snappy. In fact, our TV's inability to switch between these modes quickly is the core problem. I also suspect that this issue can be addressed by telling the Apple TV to stop switching modes so darn often by messing with (and potentially limiting) the various video modes available in the Settings app. I haven't tried but now that you mention it I might look into it.

Ideally, tvOS would be modified to minimize switching video modes (only do so while starting a new video stream/program/screensaver). The ATV home screen should always use the current mode.

I just googled the issue and this reddit thread confirms my suspicions and proposes some new settings to address it. Haven't tested them myself.