How do I change the battery in the Apple remote?

Assuming it's the Siri remote (with the glass trackpad) you don't: you just recharge it. It should take around 500 full charges and each should last some weeks even with frequent use. If the remote isn't working after being charged (you can check the charge level in the ATV settings) you need to contact Support if it's under guarantee.

The Siri remote comes with the ATV 4k but if for some reason you are using a white or aluminium remote, please see here: How to replace the battery in your Apple Remote - Apple Support