Your password is required to enable iCloud Photo Library. Open Apple ID in System Preferences to turn on iCloud Photo Library.

You may be encountering errors because you have very little space left on your device. Device storage is different than iCloud storage. iCloud syncs files across devices and still consumes space on your Mac. You'll need to delete or offload data on the device. you can also enable optimize photos and optimize files in iCloud. But, I suspect you'll still need to free up some space to make everything work properly (at least initially). I'd try to free up about 20 GB.

Click on apple icon > about this Mac > storage > manage > recommendations > click on store in iCloud a pop window appears .

Click on optimise photo library .

Click on optimise .

See the article Optimize storage space on your Mac - Apple Support

You had already optimised the iCloud storage in Mac system preferences > check marked the box of it , only photos were not optimised , also your photo box is checked , when you open photo application and its preferences these articles were already followed Set up and use iCloud Photos - Apple Support

Change preferences in Photos on Mac - Apple Support

System Photo Library overview for Photos on Mac - Apple Support

Read this article Use iCloud Photos to store photos in iCloud - Apple Support

Optimise Mac Storage: Stores smaller versions of your photos on your Mac when storage space is limited, and keeps the original, full-size photos in iCloud. Choose this option to conserve space on your Mac. To restore the originals to your Mac, just select “Download Originals to this Mac”. 

Either you optimise ( check mark it from photo preferences ) or from about this Mac location , in case if it is already checked in photo application preferences and you again check from second location it will show an error and it will prove that only an administrator has made change in the system preferences and it prompted to enter admin password , so click on ok and error will go away .

The same thing keeps happening to me. I've moved my photos library to the desktop in case that was an issue but it seems it's still taking up a lot of my storage, along with Dropbox. It says it's taking up memory on my device even though it's all supposed to be stored on the cloud

Want to get my old picture back