Remote has mind of its own

My Apple TV remote does it’s own thing when I go to move forward or back on movies, tv shows. I pause it first then move just a little to go to the point in the video I want, but the remote will decide to move on its own. I have rest it, power cycled it and anything else can think of besides buying a new remote. Still happens. Yes, it is charged and even tried it while plugged in charging.