Controller lag on Apple TV

Trying to play on Apple Arcade with a XBOX Wireless controller is nearly impossible. The controller is unresponsive and when it works it have massive input lag. I’ve updated the controller and the Apple TV (gen5) to the latest version, still it’s not working.

I feel a bit cheated since it worked great with the controller and my iPad before. Anyone else have the same problem?