Apple TV HD, tvOS 14
Verify that you are updated to the latest version tvOS, or at least tvOS 14.5, for controller support. Prior versions do not have support for Series X/S controller (model 1914).
Update your Apple TV - Apple Support
About Apple TV 4K and Apple TV HD software updates - Apple Support
Pairing and connecting are two steps. Pairing would be remembered; connecting should be done each use (session). Powering up (Xbox button with light) would likely be enough (if not connected to a previously used device), if it is anything like the 1708 model. Else go to bluetooth settings. Connecting would be confirmed with a notification on screen. It is best to connect the primary controller before launching the game, so that the game is aware of the controller from the start.
Try this for any stubborn app: