SOLUTION: iCal calendar duplicates created by iCloud

Two weeks after I migrated to Lion, I stopped syncing my iPad and iPhone through iTunes and turned on iCloud. I had 15 years of calendar data, and after the iCloud syncing, most of the entries were duplicated or even triplicated. It took me 5 days and about 5 hours with Apple tier 2 advisors to resolve the problem. The support communities had useful info, but it was scattered around. I am consolidating the useful stuff in this posting so that it may help others.

1. The important thing to understand about iCloud is that it considers itself the "master repository". Its primary purpose is to backup your data, but it cannot distinguish your data if it originates from multiple sources. My 3 devices (mac, iPad, iPhone) synced fine via iTunes. But when I connected the 3 devices to iCloud, iCloud did not continue from where iTunes left off. Instead, the calendars on the 3 devices were Merged in iCloud, resulting in duplicates. However, many repeating events (events that spanned multiple days) were not duplicated.

2. At this point, your calendar is corrupted. Don't try to fix a calendar while you are connected to iCloud (see why below). Your best choice is to delete the calendar from all platforms and then restore your pre-iCloud calendar from your archive or from Time Machine. First, disconnect your calendar on ALL devices (Apple->Preferences->iCloud->uncheck the calendar). This will delete calendar from those devices. Then go into iCloud and delete the calendar there (click the Edit button at the top of the calendar list and then click the minus button).

3. Restore your pre-iCloud calendar to your mac. Because this is pre-iCloud, all your calendars should be under the "On my Mac" section. Go into Apple->Preferences->iCloud->check the calendar. iCloud will start syncing immediately and your iCal will say "iCal - Updating" in the title bar. When it stops Updating, check your data. You will see that all of your calendars in the Mac are now in the iCloud section.

4. Add a test event and see how it updates on iCloud. Now go to your iPad and iPhone. If you did step 2 correctly, both of their calendars should be empty. Turn iCloud back on on those devices through Preferences, and the data will sync to your devices. By default, the mobile devices only sync I think 2 weeks of data. I discovered that even tho I could get 15 years of data to sync to my iPad, the iPad could not display it. So I am going to have to tolerate having only a short period of data on my mobile devices and keep the whole archive only on my mac.

TIPS I LEARNED

• Backup your calendar and address book BEFORE you go to iCloud. Use the File->Export menus to export an archive of the whole calendar (.icbu) or whole address book (.abbu).
• In addition, you might want to export each of your calendars and address book groups as individual .ics files and .vcf files. This way, if you need to combine calendars, you can simply import a .ics file into an existing calendar in iCal.
• Why can't you delete duplicates while you are connected to iCloud? I tried to do that, and just when I got close to the very end, iCloud seemed to get confused, and then it suddenly put all the duplicates back. Several hours wasted.
• If you delete the calendar from iCloud and then try to restore it to your mac, it is very important to disconnect iCloud first. Otherwise, iCloud syncs the deletion back to the mac and deletes the calendar you just restored. Just remember, everytime you disconnect from iCloud, it deletes the calendar on your device. So the sequence must be: disconnect iCloud first, and THEN restore. When you turn iCloud back on, iCloud sees this as a whole new calendar.
• I was used to the replication idea of Lotus Notes, where the newest time stamp determined which event to keep in a sync conflict. iCloud does not work that way. It doesn't use timestamps and has no idea of latest event. It just merges everything. When you first turn iCloud on, if it asks you if you want to merge your Address Book, realize that iCloud will merge at the field level. So for a contact, if you made changes in work phone on two devices, that contact will now have 2 work phone fields.
• How to restore a Lion calendar from Time Machine: In Finder, go to the Go menu. Press the option key, and Library shows up in the dropdown menu. Select Library and then open the Calendars folder. Copy everything in it to a safe place and then delete the originals. Now, open Time Machine, and it will navigate to that hidden folder. Restore everything in the Calendars folder.
• How to restore a Snow Leopard calendar from Time Machine: If you have since migrated to Lion, use the method above to get to the Calendars folder and clear it out. Now open Time Machine and go to the Library->Calendars folder and restore everything in that folder.