Apple Pay asking me to use passcode instead of Face ID

So I just got my iPhone 12 Pro a few days ago and I was setting up my Apple Pay yesterday. Even though I have Face ID set up for everything, every time I trigger the double click side button to test it, it asks me to pay with passcode without even scanning my face. No one in my family has the same issue, as after they set up Apple Pay, when they double click, it just simply scans their face. I can't seem to find a setting for this anywhere either.