I attempted to do the same with my MacBook Air 2019. Used an external drive USB-C drive to store the photo library. I also had optimize photos activated with iCloud. It took several days of background updates and I couldn't just eject the drive because it was in use. Once it settled then everything worked normally however the drive would get really hot to touch.
I then moved the library back on the MacBook Air and have not had an issue since. Those drives work great for storing and retrieving documents and even photos and videos however they do not perform well when there is near constant read/write activity.
Axel F.