Wallet App Apple Pay

When I go into the Wallet app and attempt to add a new card, I am prompted with a screen that wants me to verify a card that was previously on my Apple Pay account. It tells me to enter in the security code from the card in order to add it. I reset my phone when it had issues and have since switched banks, so I no longer have that card. I can bypass it to "Add a Different Card" but I want it to stop asking me for security code since I will never have this card again. How do I remove this from my account? It's not even attached to my Apple ID. I can't find how to remove it. Thanks.