Sounds as though it is broken if you've tested for the IR signal.
I have several white/grey remotes - some of the white ones have buttons that barely work and require very firm pressure to actuate. I think they do tend to wear out to be honest but others have no issues.
I actually prefer the older white remotes to the grey ones which seem to have less range and a narrower usable angle.
If you have an iPhone/iPad there's a free Remote app you could use provided you can use provided you can access Setting if you need to.
Apple sell the remotes quite cheaply, though you could also consider a multi-function remote or train AppleTV to use a different remote.