How/when are you trying to remove the card ? You won't be able to remove it when trying to download something, you need to just log into the account and see if the 'none' option shows : Change or remove your Apple ID payment information in the iTunes Store - Apple Support
If that is what you are doing and if you aren't getting the 'none' option : Why can’t I select None when I edit my Apple ID payment information? - Apple Support
For the card being declined, if it's a debit card that might be why. Otherwise, is the card registered to exactly the same name and address as on your iTunes account and was issued by a bank in the country where you are (and which is therefore on the account) : If your payment method is declined, or you have a billing issue with a previous purchase in the iTunes Store - Apple Support