The Server app provides for mail relaying with client authentication, but it doesn't provide for authenticated relaying over an SSL/TLS connection.
Please take the following steps to configure Postfix to relay mail to a remote SMTP server with password authentication over SSL. Substitute as required for strings in italics below. Address is the fully-qualified domain name of the relay host. The value of port is usually either 25, 465, or 587. Username and password refer to your credentials on the relay host.
In the current version of OS X Server (but maybe not in older versions), Steps 1 and 3 should be done for you when you enable relaying and relay authentication in the Server application.
1. If necessary, create or update the relayhost directive in
/Library/Server/Mail/Config/postfix/main.cf
It should look like this:
relayhost = [address]:port
2. Add these lines, above the section at the end that begins with the comment "# Mac OS X Server":
smtp_sasl_security_options =
smtp_tls_CAfile = /etc/certificates/relayhost.pem
smtp_tls_session_cache_database = btree:$data_directory/smtp_tls_session_cache
smtp_use_tls = yes
3. If it doesn't already exist, create the password file
/Library/Server/Mail/Config/postfix/sasl/passwd
with this content:
[address]:port
username:password
Here address must match $relayhost.
Then create the password database:
sudo postmap /Library/Server/Mail/Config/postfix/sasl/passwd
This action creates the file
/Library/Server/Mail/Config/postfix/sasl/passwd.db
The two password files should be readable by root only.
4. Create the file
/etc/certificates/relayhost.pem
with the CA certificate(s) to be trusted for authentication of the remote host. You get those certificates from the service provider. If you can't find a link to download them, try this:
openssl s_client -connect address:port -showcerts < /dev/null | sed -n '/-BEGIN /,/-END /p' | sudo sh -c 'cat > /etc/certificates/relayhost.pem'
The command may produce an error message that isn't necessarily significant. For servers that use the older STARTTLS protocol, rather than straight TLS or SSL, this command may need to be modified.
5. Restart the Mail service.