I had the same problem, and it seems to have been resolved although I'm not precisely sure why... here are the steps I took:
My "ghost" internet accounts flip-flopped between an Apple ID associated with a yahoo.com and outlook.com e-mail address. I could only seem to sign into one at a time even if one of the addresses was the alternate for the other.
After screwing around with it for a while, I finally set the outlook.com e-mail address as the primary e-mail address for the problem Apple ID and went ahead and signed in to that. For whatever reason, the yahoo.com e-mail address didn't re-appear as a ghost so I was as content as I could be considering my need for order. I let it run like that for a day, syncing everything, getting comfortable, and then I went to my Contacts app and found there was an iCloud group that I could not remove. When I went to menu -> Contacts / Preferences and selected Accounts, there were probably a hundred Inactive accounts associated with the problem Apple ID.
So now I went back into System Preferences / Internet Accounts and tried to uncheck the Contacts sync for the problem Apple ID, but it just kept re-checking itself. So I re-removed the problem Apple ID again. It wasn't reappearing after 10 seconds like it usually did, but I figured that was just a matter of time. Now that I knew there were a 100 ghost address books, I set out to remove them all.
I navigated to /Users/yourname/Library/Application Support/AddressBook/Sources where I found all of the ghost address books. Each one has it's one folder and in each unique folder is aConfiguration.plist file that has the associated e-mail account in the text somewhere. I went through each Configuration.plist file and for every one that had the problem Apple ID associated with it, I deleted it's parent folder. I closed and re-started the Contacts app and all of the 100 ghost accounts were gone. Also, during the 30+ minutes it took me to delete all those, the ghost account never re-appeared in System Preferences / Internet Accounts.
My theory is that those 100 or so inactive Address books kept re-populating the Internet Accounts in System Preferences.