What does the Info in the Finder show for your Library? Look at the dates for "last modified". Sometime the last modification date is way off, and then the reported size will also not be the current size.
It is to be expected, that the library on your Mac will have additional working copies, that are not in iCloud, for example the faces thumbnails, the projects, also previews for the Media Browser, the caches of the background processes.
And external editors are also caching data in library package. Sometimes these additional working copies can be as large as 25% of the total size of the original files. That is just an estimate, it will depend on the format of your original files and the external editors you are using.
To convince yourself, what Photos is storing, select the Photos Library in the Finder, then hold down the CTRL-key while clicking the library. Use the command "Show Package Contents". You will now see the folders inside the library. Just look, don't touch.
The folder "originals" is storing the downloaded originals and should be the largest item. The folder "resources" is holding most of the additional working copies and will probably be the second largest folder.