Why is the Shared Library including photos with none of the participants?

I'm trying to use the new "Shared Library" in Apple Photos to share photos of my kids with them. But it is also including many random photos of me or other scenes that do NOT include them. How does this work?