Maybe the offending HEVC has Chroma subsampling 4:2:2 (Bit depth 10 bits) or Codec ID hev1?
I have not tested how Photos.app behaves but Big Sur QuickTime Player still has the following issues:
...
There is the "old" H.264 (AVC Advanced Video Coding) and new H.265 (High Efficiency Video Coding HEVC) that needs macOS 10.13 High Sierra or later.
QuickTime Player refuses to play some H.265/HEVC flavors. Currently macOS 11 Big Sur is more forgiving but it still has the following issues with "Codec ID" and "Chroma subsampling" options.
You can check all those pesky details with apps like Invisor (my favorite because it neatly highlights differences when a folder is dropped on it) or MediaInfo.
H.265 Codec ID hvc1 plays OK (as well as H.264 Codec ID avc1).
H.265 Codec ID hev1 has an error message "This file contains media which isn't compatible with QuickTime Player" and plays audio only. There is a lossless fix if you install ffmpeg and add '-tag:v hvc1' without re-encoding in the Terminal:
ffmpeg -i input.mp4 -vcodec copy -acodec copy -tag:v hvc1 output.mp4
Chroma subsampling 4:2:0 (Bit depth 8 bits) plays OK.
Chroma subsampling 4:2:2 (Bit depth 10 bits) has an error message "This file contains media which isn't compatible with QuickTime Player". Opens anyway in Big Sur but fails in Mojave.
To muddy waters even more:
Google, Youtube, Facebook might use VP8 or VP9 video and OPUS audio codecs that might have to be converted to more common H.264/265 and AAC/PCM codecs.
Google might use webm wrapper and open source community Matroska mkv wrapper that might have to be converted to a more common mp4 (or its very close cousin m4v or mov) wrapper (usually losslessly with ffmpeg although subtitles might be unsupported in other wrappers).
VLC and IINA should have no issues with playing any decent new or old codec. Handbrake or VLC can be used to convert old codecs to new H.264 or H.265 wrapped as common mp4 (or its very close cousin m4v or mov).
