Yep, already tried all that same problem....
I really think I pinpointed the issue being that scep-helper is unable to figure out the CA certificate and send it back to the client.
The message "Key invalid" in the client can be related to anything in cert/key configuration in server.
For example if you try to enroll from "outside" you LAN and haven't forwarded port 1640, you may get this error too...
Only problem is I don't know how to check why scep_help doesn't get the root CA from OpenDirectory...
More infos. When I do an OpenDirectory archive I get a silent error in the slapconfig.log:
2011-07-30 19:06:08 +0000 4 Backing up CA certificates
2011-07-30 19:06:08 +0000 Error backing up ssl settings as hostname certificate was not found
2011-07-30 19:06:08 +0000 5 Creating archive
So there is definetly something wrong with the retrievals of the CA root form openDirectory...
I ended up rebuilding the OD (by doing standalone and back to master). However there seems to be a difference between "Server Admin" and the new Server app when creating the OD, not sure why, you would think there shouldn't, but only with the Server app I succeded creating a working OD. Since I don't have many users and I was afraid of other problems in the archive I did not use the archived OD, but instead exported the users (from the old OD) to a text file and imported them in the new OD (and losing passwords). I was also messing around with certificates but I can now confirm that profile manager is working with self signed certs. It's necessary to have all certificates (also root CA) ready and in place before creating the new OD. Hopefully this help someone else, at least this procedure worked for me.
I'd been struggling for a couple weeks with the exact same error messages and finally got it today.
I ended up doing basically the same thing that NT did and demoted OD to standalone and let Server.app handle taking it back to a Master. When I restored an archive, I noticed that all entries in CertificateAuthorities got cleared out (and resulted in the same error) so I ended up doing an export of my users, groups, and computers.
As far as which certs I used, the process that Server.app went through created a CA, an Intermediate CA, and a Root Cert. So I ditched the other certs that I was trying to use in favor of these and all was good.
All said, I think Server.app does some kind of extra magic to get everything in place to get this feature working.
I can confirm that this worked for me. Here were my steps. (This is assuming you have a signed cert, I have not tried this with a self signed cert)
1) In Server Admin (the old server app) demote the Open Directory service to standalone. This will destroy all records.
2) In the new Server app select your machine under "Hardware" in the left column, click on settings, then for SSL Cert click edit. In the new window select "None" for Certificate.
3) Navigate to /usr/share/devicemgr/backend and excute the "wipDB.sh" script as root in terminal.
4) Restart the Server app then turn on and enable profile manager. Walk through the setup wizard which will turn your open directory back on and set it as a master. When you get to the screen asking to select your cert make sure the cert you select does not bring up the yield sign error (i forget exactly what it says, something along the lines of "This cert is untrusted")
I'm not sure if all of the above steps are necessary but it's what worked for me. Hope this helps anyone else having this issue.
I followed these steps as well and the invalid key error went away but now I'm getting a timeout error on the iOS device and I'm seeing this error message in the console logs
sandboxd: xscertd-helper(xxxxx) deny file-read-metadata /private/var/folders
This is on a machine that was SL server and upgraded to Lion Server. We started running into problems with ProfileManager after the 10.7.2 update so attempted this fix above.
What's really strange is that I was able to recreate the original problem with ProfileManager on a test server and resolve the issue on the test server.
The big difference between the two servers is that the production server is an upgrade and the test server is a clean Lion Install.
Any ideas on how to resolve the xscertd-helper error would be appreciated.
1. Turn off all services under Server app.
2. Under Hardware, settings, change SSL certificate to "none"
3. Under Hardware, network, reset host name again.
4. Under Hardware, settings, change SSL certificate back to correct one
5. Turn Web service ON.
It may still say /var/empty.
6. Turn Wiki service ON
7. Recheck Web service. It should be changed to /Library/Server/Web/Data/Sites/Default.
And open port 1640 (TCP) from the internet to your server...
This is on a fresh install...
I've been havingthe same inconsistent results, sometimes able to install os x devices, sometimes iOS, but never both until yesterday.
I noticed the hostname is "server" after a clean install, not server.local (or ,private or ,fakedomain.com so I changed the hostname to server.local before creating the OD, which is when the self-signed certs are also created.
Every device in the hose enrolled first try.
Hope this helps :-)