When you try to turn on iCloud Photos on your Mac, your Mac is estimating the required storage.I t is using the worst case assumption, that all photos in iCloud are different from the photos on your Mac. So you need twice the size of your Photos Library as free storage, to be able to turn on iCloud Photos again, even if your library on the Mac is the same as in iCloud.
After a clean install the best option is to start over with an empty library on your Mac. That will be faster than restoring the library from a backup and syncing than existing restored library again with iCloud.
If you start with an empty library, there is no need for Photos to upload the library and to merge it into the existing iCloud Photos Library. You will just have to download the library from iCloud and avoid the problem with the lack of storage, you will just have to download the library from iCloud instead of the lengthy upload and merge. The added bonus will be, that you will have a newly rebuilt library, and can be sure that all items in it are compatible with iCloud.
You only need to sync your restored library with iCloud, if you have projects in your library that you still need. Projects are not synced with iCloud. Keep the copy of the old library on an externals drive, if you need to save projects.
