You're right, ExFAT is not just unstable, it actually causes corruption to Libraries. To avoid damaging the Photos Library an external drive must be formatted in either APFS format or Mac OS Extended (Journaled) format. Additionally, the drive can not have had Time Machine on it since it was formatted. There have been so many problems with using incompatible drives that the newest macOSs won't even allow a Library on a non-Mac formatted drive to open, since there is a chance of damaging the Photos database. See this:
Move your Photos library to save space on your Mac - Apple Support
Static storing on ExFAT is OK, just not running from there. So the best thing to do is to stop running that Library on an ExFAT drive. The Library needs to be put on a compatibly formatted drive. You could copy the Library to another drive (even a borrowed one,) and re-format the one you have. Or you could buy a new, sleek, fast SSD and transfer the Library to it.
If you are using iCloud Photos, then all your pictures are there, and you could just reformat the drive, create a new, empty Library, and let Photos populate it from iCloud.com.
While you mess around with this Library, you really need a dependable backup!