To be able to enable iCloud Photos, you need a system Photos Library on a compatible volume.
Open the Photos.app on your Mac, then open the Photos > Settings.
In the General tab of the Settings check, if your Photos Library is already enabled as your System Photos Library. If "Use as System Photos" Library is not yet dimmed, click it, to make your Photos Library your System Photos Library.
Then click the iCloud tab in the Photos > Settings, and try to enable iCloud Photos there. Does it work?
If you cannot even enable iCloud Photos directly in Photos, check the location of your Photos Library. Is it on a volume formatted APFS or MacOS Extended (Journaled) as it should be? Igf you keep it on an external volume, check, if the volume has been prepared as described here: Move your Photos library to save space on your Mac - Apple Support
And you will not be able to enable iCloud Photos, if you are not having enough free storage in your iCloud to hold all photos from your current system photos library.