Apple TV Remote Very Laggy when HomePod set as Default Audio Out

I have a brand new Apple TV 4k, with the new remote. When I set my default audio output to my two HomePods (original) set to a stereo pair, the remote becomes laggy and nearly unusable. The lag is only on the physical remote; using the remote app on iOS, there's no lag at all. Also, if I switch the HomePods to a temporary output, there's no lag.

Had a similar lag issue a couple years ago. The solution then was to get a faster HDMI cable. But my cable now is plenty fast. And this issue only happens with the HomePods connected. Otherwise, all is well.

Running 4.7 on ATV and HomePods.

Anyone had/fixed this issue? Some folks on Reddit have encountered it.