Ian, this is an interesting scenario, so thank you for reporting it. This is not a new issue, and it's actually not an FCP issue. It is a MacOS issue, and variations of it were reported in macOS forums in 2019:
mp4 "can’t be opened because it is from a… - Apple Community
More detail this Feb. 2025 macOS forum thread, which Luis worked on. There was speculation on that thread it might be a Sequoia regression, but on my Tahoe 26.2 machine, I can't even do File>Open With Shutter Encoder, as it either doesn't show up or it's greyed out: After updating to Sequoia 15.3, all MP4s … - Apple Community
The root problem is that the Shutter Encoder app is not properly code-signed by the developer. These commands to inspect that show errors:
codesign -dv --verbose=4 /Applications/Shutter\ Encoder.app
spctl --assess --verbose /Applications/Shutter\ Encoder.app
The codesign signature is being invalidated because of "unsealed contents" in the bundle root. It appears that his build process is placing files (perhaps a script or a readme) directly inside the .app wrapper (at the root level), rather than inside the Contents/ folder. Also, the Info.plist seems to be modified after the code-signing phase, or it is not included in the _CodeSignature seal.
There is an extended attribute on the FCP-generated MP4 file called com.apple.quarantine. That can be seen by running ls -l@ on the MP4 file. Historically, this flag was only for files downloaded from the internet. However, in modern macOS, sandboxed applications (like Final Cut Pro) often tag exported files to maintain a "chain of custody."
Because this attribute is present, macOS basically mandates: "This file has a provenance tag (Final Cut Pro). I cannot just execute it blindly. I must validate the 'Handler' (the app the user chose) to ensure it's safe to hand this file over."
In Finder you did Open With>Shutter Encoder.
Gatekeeper then inspects Shutter Encoder.app. It asks: "Is this app properly signed and notarized?"
Answer: No, therefore the open attempt fails.
Apparently, on Apple Silicon and recent macOS versions, Gatekeeper checks the interaction between the app and the File. So it's possible an update to macOS could *appear* as a regression, but in fact the problem was there all along in the codesigned app package, which macOS was not checking.
Unfortunately, the UI shows the MP4 name in the error, but it's actually a "transitive trust failure" of which the MP4 is only one element, and the improperly codesigned Shutter Encoder is the actual cause.
I will contact the Shutter Encoder developer about this.
In the meantime, as a workaround, you can simply drag/drop the MP4 from Finder to the open Shutter Encoder window.