Untrusted email server certificate problem

With self-signed certificates there are two ways to get client devices to trust them. Either you manually and normally only once trust that individual server certificate, or you do the same but instead trust the self-signed rootCA used to create that server certificate.

Since most organisations use more than one certificate it is usually much easier to distribute a copy of the self-signed rootCA certificate to all the devices and trust that on each device and then any subsequent server certificates should automatically be trusted as you would have already trusted the rootCA that created it.

You can open Keychain Assistant on your client Macs and trust this individual mail server certificate which should normally be stored in the System keychain. Similarly if you have distributed a copy of the self-signed rootCA you can find it in the same place and trust it as well. (Once the self-signed rootCA is installed and trusted the server certificate should automatically be trusted.)

There are various ways to distribute certificates including a self-signed rootCA and to automate trusting it. One of the easiest is if you are using Apple Profile Manager. You would use Profile Manager to create a 'Trust Profile' and install that on each client device. This Trust Profile can be either installed manually just by double-clicking on it, or automatically via a shell-script or by incorporating it inside an Apple Package installer, or via various Mac management tools including DeployStudio.

The exact error message may help explain things further but the server itself needs to trust its own certificate(s). So ideally the server itself would have a copy of the self-signed rootCA and have that marked as trusted.

Potentially a different issue can arise if a server is running multiple services with multiple host names e.g. mail.domain.com, calendar.domain.com and www.domain.com either multiple certificates need to be used with the correct one assigned to the relevant hostname and service or a certificate may need 'Subject Alternative Name' aka SAN fields included in the certificate so it is able to be used for multiple hostnames.

Last time I hit this, I was able to access the site and trust the certificate with Safari (and get it trusted and loaded into the keychain), but — as John Lockwood correctly comments — all of this is getting tougher, and setting up a local CA root and a certificate chain, or getting a commercial certificate from — for instance — LetsEncrypt is usually a better approach. LetsEncrypt certificates are free, though those do have to be renewed quarterly. (I'm hoping that some future version of macOS Server includes support for the ACME protocol used to acquire certificates from LetsEncrypt and other certificate providers, but that's fodder for another discussion.)

If forward and reverse DNS don't match, then mail can fail spam checks performed on remote mail servers, and — if reverse DNS doesn't match the certificate — then the associated certificate checks will (should) fail. Hosts with multiple names generally need SAN certificates.