Photo in Big Sur

Have you already tried to restart the Mac in Safe Boot Mode to clear caches? See: How to use safe mode on your Mac - Apple Support

It is is strange, that you cannot even make a new library the system photo library.

There may be permission problems with your user home folder.

To test this, try to make a library your system photo library while working from a different user account:

How to test an issue in another user account on your Mac

https://web.archive.org/web/20161114114041/https://support.apple.com/en-us/HT204443

If Photos is working correctly while using a different user account for testing, there is a problem with your current user account, perhaps preference files need trashing or permissions need fixing.

See this document on how to fix the user account:

Resolve issues caused by changing the permissions of items in your home folder

https://web.archive.org/web/20200404130045/https://support.apple.com/en-us/HT203538

What exactly is going wrong?

Can you not open your current system photo library to access it in Photos? Or do you get the error message, when you want to make the Photos Library the System Photos Library?

Where is your Photos Library stored? Is it in your Pictures folder or is it on an external volume? If you are keeping the library on an external volume, make sure the volume is compatible with macOS 11 Big Sur and Photos. Points to check: (Move your Photos library to save space on your Mac - Apple Support)

• The volume needs a wired connection. Photos 6 cannot work with a Photos Library on a network share.
• The volume needs to have the file system format APFS or MacOS Extended (Journaled).
• The volume must not have been used for Time Machine backups or have any backup software running on it.
• As a test try to check the "ignore Ownership on this volume" flag. If it is not possible, the volume is not suitable for libraries.

And you must not move the Photos Library into any synced folder, Like iCloud Drive, Google Drive, Dropbox. The syncing will damage the library.

If your Photos Library is already on your system volume, try to switch to a different, new library, just for testing. Sometimes this makes it possible to open the main library again.

• Quit Photos, if it is running.
• Hold down the options key ⌥ while launching Photos again. Hold down the key firmly until the library chooser dialog appears.
• Select to create a new library in your Pictures folder.
• Can Photos open this new library and launch? Can you make the new library your System Photos Library?

If Photos can work with a different library, quit Photos again, then double click your main library to open it in Photos. Can Photos now open it?

To repair the permissions see this page, that is incredibly slow to load - but you can see the screenshots there: https://web.archive.org/web/20190503021641/https://support.apple.com/en-us/HT203538

Here is a quote of the text only:

1. From the Finder menu bar, choose Go > Home. Your home folder opens.
2. Choose File > Get Info. An Info window for your home folder opens.
3. If the Sharing & Permissions section at the bottom of the window isn't open, click the triangle  in that section to open it.
4. If the Lock button at the bottom of the window shows a closed lock  , click the lock and enter an administrator name and password.
5. Click the Action menu  in the bottom corner of the window, then choose “Apply to enclosed items.” Click OK to confirm the action. A progress bar appears at the top of the window.
6. 
7. When the progress bar completes, open the Terminal app, which is in the Utilities folder of your Applications folder.
8. Paste or type this command in Terminal, then press Return:
9. 
   
10. diskutil resetUserPermissions / `id -u`
11. 
    
12. On U.S. keyboards, the ` character is just above the Tab key. After entering the diskutil command, if Terminal says that permissions reset on user home directory failed (error -69841), enter chflags -R nouchg ~, then enter the diskutil command again.
13. 
    
14. When the process completes, quit Terminal.
15. Restart the computer, then test to see if the issue is fixed.
16. If the issue persists, reinstall macOS.