Face id

To quote Apple:

The probability that a random person in the population could look at your iPhone or iPad Pro and unlock it using Face ID is approximately 1 in 1,000,000 with a single enrolled appearance. As an additional protection, Face ID allows only five unsuccessful match attempts before a passcode is required. The statistical probability is different for twins and siblings that look like you and among children under the age of 13, because their distinct facial features may not have fully developed. If you're concerned about this, we recommend using a passcode to authenticate.

About Face ID advanced technology - Apple Support

Does your brother know your passcode? If so, change it and do not share.

Was your brother present when you established Face ID? If so, re-establish Face ID.

Were you present in the room when your brother unlocked the phone?

In addition to Steadman’s post; is your Auto-Lock set to something “short” (e.g. 3 minutes)?

Check in Settings -> Display and Brightness.

Also, you might consider registering your face twice (Primary and Alternate appearances) to positively prevent someone else from registering their face as the “alternate.”

Dogcow makes a good point.

While the probability of your brother being able to unlock YOUR Face-ID is low, it’s NOT zero.

After taking all the “basic security” steps already posted; TEST to see if he can STILL unlock your phone.

If he can, congratulate yourselves on being a statistical anomaly and just use the passcode-only for authentication.