Extremely slow video seeking on MacBook Air M3

I noticed my MacBook Air M3 could be very slow (could take several minutes to start playback) when seeking (jump to a particular time point by clicking/dragging) while playing local videos. Playing from the start works just fine, no lagging/chopping once started (continue playback is also very slow).

• MacBook Air M3 15-inch, 512 GB SSD, 16 GB RAM, MacOS Sonoma 14.6.1
• MacBook is idle (CPU ~90 % idle, 8GB RAM used) before trying to seek, once trying to seek, both System and User usage increase to ~10 % each
• Video is a 2 hours long lecture at around 1.8 GB, 1080P resolution, MP4 format, stored at the internal drive (no slow USB drive or Samba server), playing from the builtin display
• Not every video suffers from this, only specific to certain videos (could reliably recreate for that particular file)
• Tried both QuickTime player and VLC player (Version 3.0.20 Vetinari, Apple Silicon), not much difference

I know I'm not providing much useful debugging/logging information, but I didn't find how to export log for the QuickTime player.

For the VLC player, see the "messages" from player below ("Errors and Warnings" is empty), though seemingly much useful information related to the video either:

main debug: VLC media player - 3.0.20 Vetinari
main debug: Copyright © 1996-2023 the VideoLAN team
main debug: revision 3.0.20-0-g6f0d0ab126
main debug: configured with /Users/d-fu/vlc-3.0/extras/package/macosx/../../../configure  '--prefix=/Users/d-fu/vlc-3.0/build-arm64/vlc_install_dir' '--enable-macosx' '--enable-merge-ffmpeg' '--enable-osx-notifications' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-shout' '--enable-ncurses' '--enable-twolame' '--enable-realrtsp' '--enable-libass' '--enable-macosx-avfoundation' '--disable-skins2' '--disable-xcb' '--disable-caca' '--disable-pulse' '--disable-sdl-image' '--disable-vnc' '--build=x86_64-apple-darwin19' '--host=aarch64-apple-darwin19' '--with-macosx-version-min=10.7' '--with-macosx-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk' '--with-breakpad=https://mac.crashes.videolan.org' 'build_alias=x86_64-apple-darwin19' 'host_alias=aarch64-apple-darwin19' 'CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang' 'CFLAGS=-g -arch arm64' 'LDFLAGS=-arch arm64' 'CXX=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++' 'CXXFLAGS=-g -arch arm64' 'OBJC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang' 'OBJCFLAGS=-g -arch arm64'
main debug: auto hiding mouse cursor

Free feel to ask me for more information, and thanks a lot for all inputs :)

More metadata from that video:

Kind: MPEG-4 movie
Size: 1,811,653,775 bytes (1.81 GB on disk)
Dimenstions: 1920 * 1080
Codecs: QuickTime Text, MPEG-4 AAC, H.264
Color profile: HD (1-1-1)
Duration: 01:52:44
Audio channels: Stereo