Thanks balacobaco. Didn't know about the CalendarAgent part. Without quitting that process, updating delegation was hanging forever!
This solved it for me tonight on two machines:
You must exit iCal and force terminate the CalendarAgent process in the application called Activity Monitor. When you re-open iCal the delegates slowly reappear.
Retrieving data ...