The ExFAT format is useful, when you want to transfer files and folders between your Mac and Windows PCs, but it has never been compatible with photo libraries on a Mac, neither with iPhoto, Aperture, nor Photos. Photos needs features that are not available on a ExFAT system, for example hard links, to save storage for duplicate files. And long filenames with ethnic characters may cause problems when copying files to an ExFAT drive and cause data loss. You have been lucky, that Photos has been working for you, but your Photos Library may already have become damaged on the ExFAT drive. The problem is, that Photos did not check the file system format on the previous system versions and let us store our libraries there. Now it is safer. That the "Ignore Ownership" is not shown, is a clear indication that the drive is currently not suitable for a Photos Library. Either the file system does not support it, or the drive has some backup or system software installed.
Yes, you need to reformat the drive. Reformatting will erase it, so you have to copy your library somewhere else, before you are reformatting your drive with Disk Utility. You probably have a second drive already, where you are keeping the backup copy of your Photos Library? If not, get yourself a second external drive to keep a backup copy of your library. You will need it sooner or later, better safe than sorry.
Format the second drive MacOS Extended (Journaled) or APFS, set the "Ignore Ownership" flag, then drag your Photos Library to this drive and try to open it there. And do not use the drive with your Photos Library for Time Machine backups. Plug it directly into a USB port of your Mac. It needs a wired connection, a NAS will not work.
