QuickTime TrackPad scrolling extremely slow

I love QuickTime's trackpad scrolling feature for watching replays of sports -- you can intuitively go forward or backward at any speed with continuous motion. But there's one problem I occasionally run into.

Every now and then I get a file that won't scroll normally. It is scrolling, just extremely slowly. If I scroll slowly (what would normally slow-motion or regular speed), it basically doesn't budge. But if I fling two fingers upward -- with the speed I normally would to fast-forward through several minutes in a blink -- the video moves forward just a tad, usually 2-4 seconds, maybe 5-6. Same thing happens in reverse. Both directions are capped at this extremely slow speed, which is unusable for me. I end up defaulting to the CMD+right (or left) arrow to go forward and backward, which is far less convenient.

I don't see any pattern in the files that behave this way. Almost all are mp4 files. Many I've created via screen recording, but I never change settings when screen recording, and yet only some have the issue, while most act normally. It also happens occasionally in files I've downloaded from other sources, again, with no apparent pattern. Sometimes converting the file's format fixes it; sometimes it doesn't, again with no pattern.

Any idea why this is happening, or how to fix it when it does?