I am also having this problem. The included physical Siri Remote just isn't fit for purpose; its laggy, sluggish, often disconnects, when scrolling to highlight an app it gets stuck/freezes for a second or two, then it can jump three more apps across rather than smoothly scrolling to adjacent app using one's thumb.
Its really not a great user experience for something that is meant to be defined by its user experience.
I did a factory reset more than once, no difference.
I thought at first it might be the settings, e.g the cable setting, then resolution (like on a PC), but alas it makes no difference e.g. Dolby Vision 30hz, 4K HDR10 60hz etc, (tried every combination). The picture is perfect and not the issue.
I moved the Apple 4K from a neat hidden area in a cupboard to out on display to direct line of site, 2 metres from the sofa and its makes no difference. I have the Apple TV 4th Gen is in the same place (in the cupboard) and had no issues whatsoever. This is definitely an issue peculiar to Apple 4K and the way its bluetooth remote communicates with Apple TV 4K. Be good to know what the update is on this...