Q: Photo Share Security Issue

In the past 6 months, two people who I know have been able to join a private photostream that I set up. My settings are not "public". Why/how is this happening? (In one case, it was my girlfriends ex-husband! She did not invite him, we're sure! He just showed up as a member..)

Just to be clear, I set this up in Photos in the "Shared" area. This creates a photostream in iCloud that I can invite others to have access.