Siri not responding to “Hey Siri”

So my Siri works perfectly fine except for the Hey Siri feature. I tried to setup Siri multiple times, reset my phone and disable and enable Siri, but nothing seems to work. I was playing with my friend but his headphones were broken so he was using his speakers. When I was trying to get my Hey Siri to work while on a call with him, his went off by my voice over the speaker. It’s strange because we both have the same phone (New iPhone SE). Other than that my phone is perfectly fine.