Unless you deliberately clicked a link to allow remote control of your device nobody else can take control of your iPhone (unless you are the target of certain Middle Eastern intelligence agencies, and that loophole was closed with iOS 15).
Remember your Apple Pay is only another method to access your credit or debit card. It is more likely your plastic card's magnetic strip could be skimmed and the card itself could be cloned, because Apple Pay uses a different virtual identity which is only active when you present it to a payment terminal and double click the side button to approve the payment. You should know if you had been approving payments by Apple Pay!