The most common reason why "Use as System Photo Library" would be grayed out is if your library is already the System Photo Library. Yes, that can be confusing.
Another reason the option may be grayed out is if the library is located in an incompatible location, such as on an external volume formatted incorrectly, on a network volume, or on a part of your internal volume that is being manipulated by a networked backup or syncing system like iCloud Drive(#1), Dropbox, OneDrive, etc. The safest place is in the default location in the Pictures folder in your home directory; the second safest place is on an external volume formatted as APFS or Mac OS Extended (Journaled) that is kept constantly mounted to your computer (or dismounted only extremely carefully).
#1 A Photos library can be synced with iCloud Photos, but not with iCloud Drive. Again, that can be confusing.