Apple Pay charging wrong card

For a while now pretty much every time I use Apple Pay, if I try to pay with my credit card, it charges my debit card, my debit card is set to default card, but I know for a fact many times I have definitely selected my credit card when paying and then seen it get charged to my debit card, this is very frustrating, how can I fix this?