Switching remotes between two Apple TVs, one remote has Siri, other is generic.

We have two Apple TVs, one in the living room, one in my bedroom. The bedroom TV has a Siri remote, but I don't really use it... And my mom loves talking instead of browsing, so I need to figure out how to swap remotes. The generic remote can command ether Apple TV, without being paired. It just works. But I can't figure out how to get my bedroom Apple TV to forget the current remote, I was able to have the living room TV forget it's old remote, (which got lost during our recent move so it appeared to be disconnected) but when I got to remotes and devices, that opinion isn't there. I don't want to switch the TVs, they really are mine and family, swapping all the accounts, not to mention some things work better on one better than the other. Any ideas? Thanks in advance!