That is exactly, why I am only sharing my iCloud Photos Library between my own different Apple accounts and not with other people.
When I have been sharing my iCloud Photos Library with others, I always duplicated the items before sharing them. I wanted to keep my best photos the way they were, after I have invested a lot of work into touching them up and tagging them. Any of the subscribers can change the adjustments and the metadata, and even delete the shared photos.
For me it is highly inconvenient that only our iCloud Photos Library can be used as a Shared iCloud Photos Library. My iCloud Photos Library is my most important library with my best photos. And any item we share will be removed from our personal library and moved to the Shared library, at the mercy of the other subscribers, who may have a different taste than I have. As my photos are important to me, I have only shared duplicates. But this work-around is making the Duplicates album useless, because the shared photos are shown in the depletes album.
The second problem - any shared item will appear in the shared library among my hand-picked best photos. They will be hard to find, if the other subscribers are inconsiderate and share any photo right out the camera. Before setting up the shared library I strongly suggest to have a talk with all subscribers and agree on rules of conduct.