Hi ldkerluke,
Welcome to the Apple Community.
While it doesn't seem possible to turn that music off for all photo-memories,
there are ways to make other music options available to choose, so they are ready when editing a particular memory.
Online article:
https://appletoolbox.com/change-music-memories-videos/
says under option 2:
"Only music that is DRM-free and that you downloaded onto your iPhone shows up in the Memories Music as options in My Music. So if all your music is via iCloud or streamed, it won’t show up.
That means you can only use songs without digital rights protection. Like songs you record yourself or ripped from unprotected CDs or DRM songs purchased from the iTunes Store. All Apple Music is protected and cannot be used in any soundtrack for a memory video.
_________________________________________________________
Below is Apple Support's article
:
Personalize your memories in Photos on iPhone – Apple Support
_____________________________________________________
Other option::
How to create your own albums, videos and slideshows from scratch.
Eg: https://www.youtube.com/watch?v=iRGAu70h-mw
Hope this helps :-)