Hi there AlexHongo,
If you are using iCloud Shared Photo Library with this contact, you can still add a member to your family group using these steps from Add a member to a Family Sharing group on iPhone without also adding them to your iCloud Shared Photo Library:
Add a family member with an Apple ID
The organizer of the Family Sharing group can add a family member who has their own Apple ID.
1. Go to Settings
> [your name] > Family Sharing, then tap
in the upper-right corner.
2. Tap Invite Others, then follow the onscreen instructions.
3. You can send the invitation using AirDrop, Messages, or Mail. If you’re near the family member, you can also tap Invite in Person and ask the family member to enter their Apple ID and password on your device.
If you would like to invite a new member to the iCloud Shared Photo Library, follow the 'Create a Shared Library' section here: How to use iCloud Shared Photo Library
You can also create and use Shared Albums outside of Family Sharing and include any participants you prefer with this article: How to use Shared Albums in Photos on your iPhone, iPad, and Mac
We hope that helps!