I sat on the phone with Apple Technical Support for 2 hours 25 minutes last night trying to figure this thing out. I, too, did a factory reset (at the recommendation of Apple due to a battery life problem I was having) and after both a restore from an iCloud backup and a reset "as new" I still cannot add my cards back to Apple Pay. I even tried adding a card that was not originally in my Apple Pay wallet to no avail.
They did a screen share session with me and downloaded a bunch of logs from my computer (Mac Mini with OSX Yosemite) after trying to perform a restore through iTunes. I was told they'd get back with me in a few days but to also be prepared to be issued a new phone.
I don't claim to know the internals of all the "connected" services Apple incorporates...iCloud, iMessage, iTunes, Passbook (including Apple Pay), but if I were a betting man, I'd assume this has something to do with one system not talking to another, and somewhere somehow our cards have not been deleted from the "master" system, and Apple Pay sees this as adding the same card twice, or in the event you used all 8 available slots, adding a 9th card and being rejected because your limit has been reached.
FYI, I only received one email from American Express notifying me that my card had been removed from the Apple Pay service after I performed the factory reset on my phone.
Also, as an added note, has anyone ever seen where you are asked to add the card you currently have on file for iTunes and App Store purchases as your first Apple Pay card? Supposedly, you are supposed to be able to just enter the 3 digit security code, and it is automatically added to your Apple Pay wallet. I have never been provided this option, even though my card on file is my Wells Fargo checking account debit card...an approved card for the Apple Pay service.
I will report back if Apple Tech support comes back with any positive insight into this debacle...