As @ibrennan indicated, I had to rm ~/Library/Application Support/AddressBook/Sources/*
I think I found the original cause of the problem on my system: prior to updating to Yosemite, I backed-up my address book to a Contacts Archive (.abbu file). After updating to Yosemite and setting up a clean iCloud account, I imported the Contacts Archive file (MyAddresses.abbu) into the Address Book. Apparently, the Contacts Archive contains information about the internet account in which the contacts originally resided--In my case, this was another iCloud account. This process resulted in the undeletable "Unknown Source" account in Contacts.
The solution I found was;
- go to another Mac that had my entire address book and use the "Export vCard" feature to export the entire address book to a .vcf file.
- Clean-up the new Mac's Contacts (rm ~/Library/Application Support/AddressBook/Sources/*) and quit Contacts app.
- Go to System Preferences->iCloud and enable the "Contacts" checkbox
- Launch Contacts, select "All iCloud" in the left panel, and import the .vcf file
This resulted in correct account setup, but I did lose all my email groups (they apparently dont't come along in the a .vcf file format)