I've had this happen on two different MacBook Pros (but not my iMac), on both Catalina and Big Sur. I think it might have something to do with external monitors being attached.
In any case, while I can't *prevent* it from happening, the fix I just found on 9-to-5 Mac works perfectly for me.
Essentially, when you hit a video playing too fast, open up Activity Monitor.
Near the top there should be a "coreaudiod" process, using somewhere between 5-20% CPU, and with a "CPU Time" value in multiple hours. If you don't see it there immediately, you can use the "search" button in the toolbar to search for "coreaudiod". Verify that the CPU is non-negligible (ie, above 1%).
This is the culprit.
"coreaudiod" is a background service which handles audio on the Mac; because it is a background process, it is managed by the OS itself so if it "goes away" it will simply be restarted.
Okay, in Activity Monitor, click the "coreaudiod" process, then click the "X" in the toolbar, then "Force Quit" (in my experience with this issue a 'Quit' is simply ignored by coreaudiod when in this state, so 'Force Quit' is necessary). On a recent fast Mac it may look like nothing happened, but if you look closely the "CPU Time" column for "coreaudiod" is not in the seconds; this is because the service automatically restarted when it was killed.
After doing that, try playing your video again. If the problem you are having is the fairly common one I was having at least, this should have fixed things, without restarting your Mac.