How/when are you trying to remove the card ? You won't be able to do so when trying to download something, you need to just log into the account : Change or remove your Apple ID payment information in the iTunes Store - Apple Support
If that is what you are doing and it's not working have you tried using your computer's iTunes instead of doing it on your phone (or vice versa) ?