Apple TV 4k remotes drains battery fast

It seems this has been asked before, but the solution of resetting and pairing remote again has not worked for me. The issue first started with my original remote and the first symptoms were related to volume control being out-of-control. I realised this kept happening when battery on the remote was very low. So I charged it and it worked fine for a short while.

I gave up and bought a new remote, this one with USB-C charging. I have been using it only for a couple of weeks, and now the same thing is happening again.

So it seems the fault may not be with the remote? I actually have 4 ATVs, and only have this issue with one of them.