By default the iCal service is polling an IMAP inbox for invitation replies every 30 seconds. This is what produces the error message you quoted. In my case it was also preceded with the following message:
[mailgateway] [IMAP4DownloadProtocol,client] Unhandled unsolicited response: ['OK', ['HIGHESTMODSEQ', '1'], 'Highest']
The settings for this are found in the file /etc/caldavd/caldavd.plist when searching for the first occurrence of <key>Receiving</key>. In my case the server’s address was set to localhost and SSL was turned on. Since the mail server uses a valid SSL certificate, connecting to localhost with SSL is bound to be problematic.
Setting the value for <key>UseSSL</key> underneath <key>Receiving</key> in /etc/caldavd/caldavd.plist to <false/> and restarting the iCal service stopped the error message for me. You might also want to turn off SSL for sending invitations. See underneath <key>Sending</key>.
Then I tried to change the settings so that connections are not made to localhost but to the server’s actual DNS name (e.g. server.example.com) and turned SSL back on. After seeing that the mail server refused to login the com.apple.calendarserver user, I changed the account field for the Keychain item icalserver.invitations that read com.apple.calendarserver@localhost to end with server.example.com. Unfortunately that brought back the original error messages, so I had to revert to the first solution of just turning off SSL for iCal’s mailgateway. As both the Mail and iCal service is running on the same machine this is no problem.
You also mentioned that your users are seeing repeated password dialogs. If you use an SSL certificate for your iCal service you can try to turn off Digest authentication and instead use Basic authentication. This means that passwords will be sent in the clear, so again make sure that communication is encrypted using a SSL certificate. At the top of /etc/caldavd/caldavd.plist change the two values for <key>Enabled</key> underneath <key>Basic</key> and <key>Digest</key>. Restart the iCal service after making this modification.
Thank you for posting your solution. I was beating my head against a wall until I came upon you post. No email invitations were being sent out to non system users. Internally, everything was working fine, but I just couldn’t get emailed invites to work. I did the things you described above, especially turning off SSL. There were some additional things that I needed to do. I had configured DNS differently than the simple DNS that the server sets up for you. By default it sets up a primary zone of server.example.com. I needed to change the zone to example.com and add an A record in that zone for the server. I also set up mail to receive mail for example.com vs. server.example.com. This was important because when you turn on “Allow invitations using email addresses” the default email address is firstname.lastname@example.org. This didn’t work as it needed to be changed to email@example.com.
I do have email invitations working, but there are still problems. I could not get the invitations to work with an outside mail server, with SSL on, or using a different account than the Apple provided one. Even though it’s now working, my logs are still filled with the following:
2012-02-20 11:06:17-0500 [-] [mailgateway] 2012-02-20 11:06:17-0500 [IMAP4DownloadProtocol,client] Unhandled unsolicited response: ['OK', ['HIGHESTMODSEQ', '1'], 'Highest']
2012-02-20 11:06:17-0500 [-] [mailgateway] 2012-02-20 11:06:17-0500 [IMAP4DownloadProtocol,client] Unhandled unsolicited response: ['OK', ['URLMECH', 'INTERNAL'], 'Mechanisms', 'supported']
This is true whether or not I use localhost or my servers FQDN in the mail setup for invites.
Any more thoughts on how to stop these errors?