How do you unpair or turnoff the Apple TV Remote?

When you have an Apple TV Remote (with the white ring around the menu button, and the siri button)

and that remote has a damaged button/touchpad which transmits ghost touches once in a while,

how do you permanently unlink (unpair, or turn off) that remote?

I don't what the solution where it will reset, unpair and then pair immediately like when holding menu+vol-up for 6 sec..

I want the remote to be forgotten by the Apple TV so that I can use my iPhone's Apple TV remote app without the Apple TV Remote sending ghost touches.

tvOS 14.0.2

In Settings -> "Remotes and Devices" , for the Apple TV remote, there is no "unpair remote" in the settings if the remote is currently paired. The remote's menu only shows the remote FW version, serial number, battery charge. But no option to unpair.