Magic Remote won't pair with AppleTV 4K

My magic remote won't pair with my AppleTV 4K anymore. It keeps saying "Bring it closer to your Apple TV to continue pairing" without any feedback when bringing it close. Even after waiting for hours. How do I fix this?