Face id

As above:

Tell your brother to create a new Face ID, and delete any existing faces.

When establishing the new Face ID, do it in a room with no other people.

When using Face ID, do so in an area where no other faces are visible.

Tell your brother to create a new passcode, and do not share it.

You seem to be concerned about safe access, but go to great lengths to defeat security protocols.

If after taking those measures, it still is not satisfactory, contact Apple.

Tell your brother to change his passcode, and not to share it.

Tell your brother to re-establish Face ID.

If you insist on sharing your passcodes, yes you can unlock the other person’s phone.

Yes, it will continue to do that, because you know each other’s passcodes. If you are concerned about someone accessing the device, stop sharing the passcode. By doing so, you are telling the phone “it is ok if this person opens my phone”.

Do you know the passcode to his phone?

Were you in the room when your brother established Face ID?

Was your brother in the room when you unlocked his device?