What has changed before you could no longer open your Photos Library? Did you just install a system upgrade or move your Photos Library to a new external volume?
The error message is typical for a Photos Library that is stored on an incompatible volume. If your library is on an external drive, make sure the volume is prepared as described here: Move your Photos library to save space on your Mac - Apple Support
The main issues are:
- The file system format of the external volume needs to be MacOS Extended (Journaled) or APFS and not case-sensitive.
- The drive needs a wired connection, it must not be remotely mounted on the local network, even if it seemed to have worked for quite some time.
- The volume with your Photos Library must not have been used for Time Machine backups or have other backup software installed.
- Don't use the folder Users/Shared for your Photos Library or use any folder that is synced by some cloud service, for example iCloud Drive or Drop Box.
- Don't use a removable storage device like an SD card or USB flash drive, because that may cause data loss.