Your assumptions are right on.
léonie is warning you about some possible problems with having a large Library. Having a cat (follow me here) that's the same size as its cat carrier is a problem, because the cat needs to turn around. (Here's the point:) A Photos Library also needs turn-around room. As it makes copies for transferring and other operations, it takes up way more space than the static size of the Library might suggest. Usually, you would like to have as much empty storage as the size of the Library package, itself.
Another problem with a large library is that some operations become very slow. For many operations, this slowing is not linear--that is, twice the Library may take way more than twice the time for some functions. Obviously scanning for faces, categories, and other such properties takes longer, as well as searching for those things. But even editing and displaying pictures may take longer. Updating the Library when the OS changes will take longer.
If somehow the large Library develops a problem, the cause may be harder to find and the fix harder to work. A single corrupt file, for instance, may cause the Library to stall, and make fixing it hard.
Backing up a large Library may become so time consuming that you are tempted to do it less often.
And if something awful happens to a large library, it happens to all your pictures.
What many of us do is divide our pictures up among different Libraries. Photos lets you have lots of different Libraries, but only one can be the System Library--that's the one that synchronizes with iCloud. You can switch between Library my just double clicking on the next Library package, or you can option-click on the Photos app and it will give a list of all the Libraries you can switch to.
So, for instance, I have libraries for different trips and special events-- these don't change often (only when I go back to them to sort and edit them) so they don't need to be in constant sync with my other devices.
But I also have a Favorites & Recents Library, my System Library, where I put the Favorites from the other Libraries that I really want to see on my phone or iPad. And that's where the new pictures from my phone are synced to. Then I periodically transfer those pictures from my System Library to a Libraries that are more archival.
My System Library is small enough to reside on my MacAir, so I don't have to worry that unplugging could possibly cause problems. I use the app PowerPhotos ($30) to make copying and switching and searching through libraries easier.
I think this is the sort of stuff that léonie was getting at...