Your Photos library is incompatible with iCloud Photo Library.
I have now got Big Sur 11.01. I have had problems for years with Photos and iPhotos including being repeatedly asked to open Photos - even when it looks like photos is open on my desktop! I cant make any photobooks or store to iCloud. I seem to have a Photolibrary 2 in users/(me)/Pictures and when I hit Other Library, I have found a few (about 80) photos in Pictures. they are greyed out and wont open when I tap 'open', they also seem to be in the users/(me)/Pictures folder.
when I tap the Photos icon in the Dock - all my pictures show up.
Help - this whole business of Photos, iPhotos etc is driving me loopy
MacBook Pro with Touch Bar