Have you already tried to open your Photos Library directly? Double-click the Photos Library (not the Photos.app) to launch Photos on this library.
If Photos still cannot open the library, test if Photos can run at all, by creating an additional, new, empty library for testing.
- Open your Applications folder and hold down the options key ⌥ while double clicking the Photos.app in Applications (not the Dock icon or an alias). This way you are sure to laugh the Photos 6 app installed by Big Sur and not an old version.
- When the library chooser appears, select to create a new library. Can Photos launch correctly on the new library?
- If yes, repeat by selecting your current library. Does it work?
If Photos can still not open the library - Where is your Photos Library stored?
Are you keeping it in your Pictures folder (where it is stored by default) or have you moved it to a different location? If you have moved the library, check, if it is in a compatible place - the library needs to be on a locally mounts volume, not a NAS, not in a synced cloud folder, and the volume must not have been used for Time Machine backups.
The file system format should be APFS or MacOS X Extended (Journaled)
Also, consider to install the recent updates. The current system version is macOS 11.4.