OK, I thought I would weigh-in since I've borrowed everyone else's brain cells for a while now. Thanks for everyone's help, including the links, but nothing worked until now.
I have a fresh install of Leopard Server 10.5.1 and I have an installed self-signed cert (through Server Admin) that I converted to a signed cert using a self-signed Certificate Authority - CA - (built using Keychain Access/Certificate Assistant). I have installed the CA's cert export into both Leopard and Tiger email clients and they can connect to my Intel Xserve to both securely download (IMAP) and send mail via SMTP. I don't know how this will help for folks with pre-signed cert problems.
+Note: I did modify postfix's and amavisd's configs as per the tutorials at
http://osx.topicdesk.com/content/category/4/18/62/ concerning 'Front Line' spam techniques and 'Virtual Mail Users' to even get things working. Those guys deserve a big fat scrumptious holiday cookie for their work.+
*This was not easy.* How do you ask Apple for 5 hours of you life back? As with several of the configurations under Leopard Server, things seems to be broken. However, the way I got it to work makes sense. So this is not really a work-around.
Here's what I did:
1) If you have made previous attempts to sign a cert made through Server Admin you will need to delete any associated files. CA files are in ~/Library/Application Support/Certificate Authority. Certs can be deleted via Server Admin, which will remove them from Keychain Access (only System Keychain) and /etc/Certificates. If you have made it to a successful sign of a cert, but it didn't work, you will need to delete any certs in your login or similar keychains you have made.
2) Create a new self-signed cert under Certificates in Server Admin. *Do not* give the private key a passphrase. Under Mail, apply it to SMTP (set to USE not REQUIRE) and try out the connection from an email client. You should be asked to accept the cert as from a trusted source. If not, there's something funky with Server Admin and you the rest of this won't help you.
3) In Keychain Access you should notice the cert you made, and its keys, in the System Keychain, however it will have a red circle with a white X denoting it is not trusted yet. Do not do Get Info on this and set it to Always trusted. The process of assigning a CA will do that. Create a new keychain specifically for your CAs and certs.
4) Create a fresh CA under the Keychain Access/Certificate Authority menu or launch Certificate Assistant directly (/System/Library/Core Services/). Make it your default, if needed. I was not able to sign the cert I made with a CA made without overriding the defaults (check the override button). The CA has to dictate to the cert what it is allowed to secure and how, and the default CA only does email authentication and signing, what the 'S/MIME (Email)' setting apparently does (as far as I can tell). The certs made with Server Admin appear to be fall under the 'SSL Server' setting for a CA. You can check that by pressing the 'i' button when the cert is selected in the System Keychain (under Key Usage).
In the 'Key Usage for This and Users of the CA' dialogs I unchecked Critical and checked capabilities 'Signature, Key Enciph, Data Enciph, and Cert Signing'. In 'Extended Key Usage' dialogs I checked 'SSL Server Auth' only.
Leave 'Basic Constraints for CA' as is - Included and Use Cert as Cert Auth checked. Skip 'Basic Constraints for User of CA'
If you try to skip 'Subject Alternate ...' (for CA or Users of CA) it will throw an error noting you need a domain and/or IP address. Check the Include box and input your mail server's fully qualified domain name and the IP it is bound to.
Finally, check 'On this machine...trust ... this CA' and save it to the new keychain you made. Then drag just the cert into the System Keychain (not the keys). I'm not clear on why to do this, but if you save it in System first, you may never be able to export any private key. System Keychain seems to have its own
private password!
Your CA should now be the right kind of CA for signing your cert. If not you will need to play with the setting on Key Usage, which means going through this whole process repeatedly. 😟
Quit/relaunch Keychain Access (
important!). Otherwise you will not be able to assign the CA to any new certs.
5) Gerate a CSR from Server Admin for your cert. I chose not to do the email thing and dragged the icon (a text clipping) directly onto BBEedit, which made a nice clean text file (MacRoman with Unix line endings) that I saved as 'my.cert.domain.csr'.
6) Launch Certificate Assistant (via Keychain or directly) and choose 'Use your CA to create a cert for someone else'. Then drag your .csr file onto it. Choose your CA and make the CA default. Don't override and defaults on the first try. You can mess with that later, if successful. The Assistant will place the new cert into your default keychain (mine is still 'login'). Export the cert,
not similarly named one from the System Keychain, into the .pem format and save it along side of your .csr.
7) Open the .pem file in a plain text editor and copy the cert text to your clipboard. In Server Admin choose 'Add Signed or Renewed Cert from CA' for your cert in the list and paste the .pem cert text. Click Save to commit the change. If under the Authority column your listed cert shows your CA instead of 'Self Signed' then it worked. Yeah!!! If not there is some funkiness along the way, and you're going to have to experiment.
Now, here's the Catch-22, or the definite bug present in Server Admin's routine. If you assign a passphrase to your private key when you make your Server Admin cert initially, everything seems to be OK. But (at least on my system), Postfix fails to be able to read its file (/etc/certificates/my.domain.key). My errors were:
+warning: cannot get private key from file /etc/certificates/<my.domain.here>.key+
+warning: TLS library problem: 1878:error:0906406D:PEM routines:DEF_CALLBACK:problems getting password:pem_lib.c:105:+
+warning: TLS library problem: 1878:error:0906A068:PEM routines:PEM
doheader:bad password read:pem_lib.c:401:+
+warning: TLS library problem: 1878:error:140B0009:SSL routines:SSL
CTX_use_PrivateKeyfile:PEM lib:ssl_rsa.c:709:+
+cannot load RSA certificate and key data+
Not good, to say the least. So make your Server Admin cert without a passphrase (there's some bad advice). That got rid of the Postfix errors. Try it with a passphrase and see if you get the SMTP errors. I was able to make separate certificates using Certificate Assistant under my CA (using it as a template from ~/Library/Application Support/Certificate Authority/MyCA/MyCA.certAuthorityConfig - as described at
http://www.afp548.com/article.php?story=20050813004038282 )
however if I tried to import the separate cert, its keys and CA, without a passphrase for the private key, it
always failed to import. With a passphrase I was stuck with the Postfix errors. So the ONLY way I could get this to work was as described above - a Server Admin generated cert with its .csr signed by a CA.
Now I know what 'Trust is earned' means. 🙂
Please let me know what I am doing back-asswards, or if you have found a better way.
Larry
Message was edited by: Larry_S
Message was edited by: Larry_S