I had the same problem and it was driving me crazy. I even did a full restore to factory settings and that didn't solve it. I turns out that I had another remote that went to the television (out of the 6 or 7 remotes I have for different things) and it was underneath something that was pressing some buttons down. Once I got it out and it wasn't being pressed anymore, the Apple TV remote worked normally again. So I'd say that this issue is caused by some kind of interference with the signal the Apple TV remote is sending. In my case it was another remote for a different device being pressed on. I'd check around the room and think of what might be sending a signal that would conflict with your Apple TV remote.
I have this problem when scrolling horizontally or vertically sometimes it will jump further than I want it, most annoying when trying to type text for a search.
This problem orcurred on my Apple TV 2 after owning it for awhile, when the Apple TV 3 came out and I got one the new remote was working nice again, but over time, its the same problem. I'm thinking the remote just wears out over time. Probably from my kids dropping it on the hard wood floor.
I'm thinking of picking up a new remote at the Apple store to see if that fixes it. I'll update if my issue boils down to just a glitchy remote.