Make Siri Only Respond To Your Voice On Homepod?

So, I got the Apple Homepod along with two Homepod Mini’s for Christmas. I really love them. However, I have an annoying issue that I can’t seem to find a resolution for. While I know that I can restrict Airplay to only myself in the home, I can’t find a way to make it so that no one else can change the music by saying “Hey, Siri” themselves.

This is very annoying if I have guests and am playing music and some annoying guest keeps saying “Hey, Siri” and changing it often. I can’t seem to find a way to make Siri only respond to my voice. Is there a way to do this? Again, I’m not referring to AirPlay, or controlling Homepod from the iPhone.