Today I reloaded the server in question. Downloaded 10.7.3 via the Recovery Partition, restored the OD archive, and tried to set up iCal. I have exactly the same problem.
I set up a calendar on a Lion client, and try to set it to share with someone who has an OD account. The sharing dialog closes, but no invite is ever sent. If I try to share with someone who is not in OD, I get error messages in the server log.
An entire afternoon of Googling shows a lot of people are having the same problem. Best I can tell, it seems to happen if your e-mail server is hosted elsewhere, as if iCal Server assumes that the mail server is on the same Mac. This is not the case with this setup. Their web page and domain are hosted outside as well, but we have a DNS entry made so server.domain.com forwards to the Mac Mini server inside their office. We've tested this for VPN, Remote Desktop, and FileMaker access, and all work fine.
I've tried two types of configurations, neither of which is working.
1. Created an ical@domain.com account with their mail host. I used that outside mail server information for IMAP / SMTP settings, but every time I try to share a calendar the iCal error log shows IMAP login errors. Their mail server does not use SMTP authentication. They have the option to use SSL or not, but neither work.
2. Used com.apple.calendarserver@server.domain.com, and used the default localhost server info. I then tried to set up the server's built in Mail server to forward mail to their outside mail host. This started giving me
[mailgateway] 2012-04-05 19:51:05-0400 [IMAP4DownloadProtocol,client] Unhandled unsolicited response: ['OK', ['HIGHESTMODSEQ', '1'], 'Highest']
I now get that error no matter what I do, meaning something broke again.
I've gone through several Google pages referring to greylisting, the postfix, etc, but since the server is not running a Mail server, none of these apply.
I have no idea what I am doing wrong. Lion has been out a year, surely in that time this has been worked out. So why can't I find a solution? Everywhere I look it's either a thread starting out with the same problem I am having, followed by a dozen or so "me too" posts, or it's someone saying they managed to make it work, but their fix doesn't work for me. I'm really frustrated right now, and so is my customer.
Oh, almost forgot... now we have a new problem. FileMaker Admin requires access to port 16000, but for whatever reason the server is blocking that port. It's not running a firewall. Good to know that reloading causes more problems than it solves.