Remove picture from automatic Memory Video

Ok. So my Mac have created e memory video. Awesome!

However, I want to remove some pictures from that video (ie, only from the video, not from the photo Library).

How do I do that!?

The obvious choice, by pressing backspace, says that it will delete the picture from iCloud and all devices. The official help page also mentions the ability to remove pictures, and I'm quite certain I have done that before... 🤔

So now my son will get a a birthday video with some crappy non relevant pictures in it, that doesn't belong there. Not so funny.

Please advise.

macOS Ventura 13.0.1, Pictures v8.0