iwatch apply pay double click

Hey, I’m struggling with apply pay on my iwatch SE. I have set up the card, called the bank to get it approved which all went well. But when I double click the side button it doesn’t take me to the wallet, it takes me to the last app I was in! When I manually go into the wallet in apps the card is there and it tells me to double click to pay but when I do it takes me back to the last app I was in!! What am I doing wrong?