The iCloud.com EOL bug can cause Notes field replication

Question

Level 3

869 points

The iCloud.com EOL bug can cause Notes field replication

This is purely informational. I think very few people will be affected by this bug. If you're seeing replication of the content of "Card" Notes fields though, it might help to know about it. I don't know of a workaround yet, but I'm still studing the problem.

I discovered it when I migrated from MobileMe to iCloud two days ago and found that several hundred of my Contacts/Cards had extra line feeds (blank lines) in the Notes field text, and that with each synchronization the content of the Notes field was being replicated.

I believe this is the cause (excerpted from a longer blog post) [1]

When I inspected my iCloud "Cards" I found that not every Card/Contact Note field had problems. The ones with problems not only showed duplication of Notes content, they also showed extra spaces between paragraphs. The problems were also restricted to Contacts/Cards that started life in Microsoft Outlook, and that moved via MobileMe to OS X.

This smells like a a new line definition aka line termination aka line break aka end-of-line (EOL) marker bug. One of the most intractable and maddening bugs in the history of software (it's been a curse, for example, for Blogger.)

Windows uses CR+LF to define an 'end of line' - or paragraph break. Mac Classic used CR. OS X uses LF. My Address Book contains contacts that started out in Outlook, then went via MobileMe Windows Control Panel to live in MobileMe, and then into my desktop machine.

MobileMe could handle the CR vs CR/LF variation. So can Address Book in Lion and in Snow Leopard. It appears, however, that iCloud.com cannot handle a mixed set of Contacts with variable EOL markers.

To make matters worse, because iCloud.com and OS X are managing the EOL encoding differently, each time my iCloud contacts sync with a new machine the Notes field is treated as non-matching, so the contents are appended. This repeats until a Notes field maximal length limit is hit.

It's a nasty bug. I'll be experimenting with ways to work around it, such as exporting an Address Book archive, then reimporting as UTF-8 or Windows or Mac rather than "automatic". Perhaps Address Book will do the conversion for me.

Ironically, I came across this line feed distinction when I pasted the above text in from a web page source. I had to add line feeds to define paragraphs.

The CR/LF vs. LF/CR vs CR vs LF schism in software history gets my vote for worst schism ever.

[1] http://tech.kateva.org/2012/06/icloud-transition-went-as-expected.html

Posted on Jun 28, 2012 6:55 AM

Reply

Answer 1

kiddo23

Level 1

12 points

Jan 22, 2017 3:23 PM in response to jfaughnan

Thanks jfaughnan, for your detailed explanation of the bug, and your commitment to finding a solution, but the most commendable of all is your kindness in freely sharing what you know and have learned with the world. I noticed your last post was in mid 2012, yet I am still seeing this problem now. We are talking about 5 years in between, and amidst all the ios updates, macos updates, new iphones, ipads, watch models being released during this span of time, Apple never managed to fix this bug? I really hope I am wrong about this, because it's gross negligence of Apple.

Reply

Answer 2

jfaughnan Author

Level 3

869 points

Jun 29, 2012 7:57 PM in response to jfaughnan

I'm making some progress on a workaround.

I found that exporting a vCard exposed the extra line spaces; the vCard has twice the \n as expected.

I also found that taking the text on a round trip through TextWrangler resolved the EOL bug. When I paste back from TW into Address Book then export vCard the \n count is as expected.

I wonder if I can write an AppleScript to use TW to translate the EOL characters by this round-trip copy/paste method. Crude, but probably effective.

Reply

Answer 3

jfaughnan Author

Level 3

869 points

Jul 3, 2012 8:15 PM in response to jfaughnan

I have a candidate fix for this bug. It's a bit of a kludge that relies on Bento, Address Book, TextWrangler, and Contact Cleaner.

I'm still testing the resuts, but early tests look good.

Details on my blog, excerpted below:

http://tech.kateva.org/2012/06/icloud-transition-went-as-expected.html

pdate 7/3/2012: vCard export/import won't work, but a hybrid hack looks promising

Many fields are omitted from Address Book's vCard option. So I can't do clean up and restore from there, I'd lose too much. Worse, Address Book Import is also quite weak. Instead on a test machine running Lion and Address Book 6.0 I tried this convoluted process on a copy of my Snow Leopard Address Book.

I selected all contacts and exported Group vCard (1820 contacts)
I used Bento to delete all Notes (select column, hit delete, exit Bento)
I used TextWrangler to replace \n\n with \n everywhere and remove a trailing \n at the end of the NOTES string. This removed some paragraph definitions but I didn't mind that. Simpler to do.
I then imported the vCard. Address Book said it has found 1818 duplicates and merged those in; I ended up with 1822 Cards. So there were two duplicates.

The results seemed good, I no longer found duplicate notes. The Address Book synchronized to iCloud far more quickly than before, and iCloud also appeared fine and without duplicate Notes.

I considered using Address Book merge to resolve the two duplicate Cards, but it is too aggressive and would merge addresses I wish to keep separate. So I used Contacts Cleaner. It found 3 duplicates and a (new) 5-6 duplicate addresses. Those were quickly resolved and I ended up with 1819 cards.

This approach, though inelegant, appears to work. I'll do some more testing before I try it on my real Address Book. (I haven't tested yet to see if Snowie will open a Lion Address Book. I suspect not, but the process I used will probably work on Snowie. As long as one backups up the Archive, it's easy to restore an Address Book.

Reply

Answer 4

jfaughnan Author

Level 3

869 points

Jul 4, 2012 1:58 PM in response to jfaughnan

I had a fix candidate -- but then I disovered that my Groups were metastasizing and "poisoning" every machine that I connected to my iCloud account. Irreversibly. Even when I removed the iCloud account.

Details at my blog post [1], but this is what my Groups look like in Address Book and iCloud:

Large scale replication of Groups. No way, short of hours of click and delete, to eliminate them.

I'm done. iCloud and Lion aren't really ready. It's not the bugs (though they're bad enough), it's the lack of recovery and troubleshooting tools - the iCloud equivalent of MobileMe sync services reset. It's as though Apple though they'd get everything right -- despite their MobileMe history. (Ironically, MM worked well for me over the past six months.)

I'll try again when Mountain Lion is out.

[1] http://tech.kateva.org/2012/06/icloud-transition-went-as-expected.html

Reply

Answer 5

jfaughnan Author

Level 3

869 points

Jul 27, 2012 8:41 PM in response to jfaughnan

Mountain Lion didn't fix anything, but it's much faster than Lion at iCloud sync.

What did fix things were two AppleScripts.

One is from Nigel Garvey of MacScripter, it fixed the line terminations.

http://macscripter.net/viewtopic.php?pid=153557#p153557

The other was a simple script I wrote to remove the replicated groups:

http://tech.kateva.org/2012/07/icloud-group-replication-resolved-by.html

I ran Mr Garvey's line termination fix script against my Snow Leopard Address Book, then used the Group delete script to clean up iCloud, then imported into iCloud. Seems to work.

Reply