EXIF data not updating when "Importing: Copy items to the Photos library" is disabled

I just imported over 35k photos which I have on a cloud drive. Since I do not want them copied to the Photos library (iCloud), I unchecked "Importing: Copy items to the Photos library".

This works more or less fine as long as I import only a few thousand pictures at a time (else the Photos app crashes).

I created an intelligent album to find all pictures that are NOT geotagged, and I am using some external tool to tag them.

This is working fine, however the Photos app never rescans the files that are changed on the filesystem, so EXIF Data is never updated. I was hoping that unchecking "Importing: Copy items to the Photos library" would not cause this kind of issues, since the true data is on the FS and NOT in some internal DB of the Photos app.

Is there any way to update/rescan all photos from the library without having to delete everything and reimporting? For 35k files and a Photos app that crashes or takes ages for importing it is just not the way to go...