How to make QuickTime the default player for all video formats? - .m4v, .mp4, .mov, etc

Recently upgraded to MacOS Big Sur. How do I make QuickTime the default player for all video files? When I open an .M4V file, it opens in the Apple TV application which does not support scrubbing with Magic Mouse.

When I right click and choose "open with" then "other" and select Quicktime as default player, it only applies that to the specific video file I am trying to view - not all .M4V files

How do I make QuickTime the default player for all video files across the board?

Thanks!