Siri is responding to everyone’s voice

Correct. There is no specific voice identification, only phrase / command recognition.

As long as its a command Siri can respond to, it does not matter who says it, it will respond.

Though generally, Siri will only respond on the device that is closest to the person saying the command or was the most recently used.

See here: How 'Hey Siri' works with multiple devices – Apple Support