Try the steps I posted here: https://discussions.apple.com/thread/3222240
Note that with iCloud (and I guess any other source) there is a Sources folder inside of the AddressBook folder. It has the same indexing databases. At this point, I don't know of you would need to remove those, also. When you throw away the iCloud one, it will take a few moments for the iCloud contacts to reindex.