OS X Server - Mail - is this setup secure? smtp blacklisting safe?

Your mail server is public, and it takes a minute or two for a single site to scan the entire active parts of the IPv4 address space looking for mail servers. The botnets can scan for mail servers even more quickly than that. Obfuscating your server host name here means that folks here test a few things, and the riffraff will still find the server and start probing it, if they've not already done so.

1: DNS on OS X is rather limited via the GUI. If you want to start setting priorities, you'll likely need to use the command line. AFAIK, Server.app doesn't allow setting MX priorities, though the underlying BIND DNS server does.

2: "Safe?" No. See above. The riffraff have already found your server, if TCP port 25 or one of the other common ports is open. How to see what's happening? Review your setup and your logs, and particularly start reading the details and documentation posted over at the Postfix web site. As for connections, I'd usually only allow TCP ports 25, 993 and 587 forwarded through the firewall, as well.

3: Accept the certificate that the server offers while on your local network, or set up your own certificate chain, and load the public key for your certificate authority into the client devices, or purchase a certificate. (There's no difference in security between an equivalent-bit-length private cert and a purchased cert, either. The difference is that the purchased cert has the associated root certificate already loaded into your clients — you need a trusted way to do that with your own certificates or with your own certificate authority.)

For your mail server, your local DNS forward and reverse and your public DNS forward and reverse translations must match for your mail server to be accepted as legitimate by other mail servers. That is, the IP address returned by a dig command must — when fed into a dig -x command — return the host name you started with. The following shows an example host name and an example address, and attempts to illustrate the nature of a matching DNS translation for the host name. If the dig -x gets you something other than mail.example.com, your mail server will not be accepted by many other mail servers — it'll be assumed to be a spam engine.

$ dig +short mail.example.com

192.0.2.10

$ dig +short -x 192.0.2.10

mail.example.com

$

To dump out your changed settings:

postconf -c /Library/Server/Mail/Config/postfix -n

Usual key settings here — there are various discussions in the forums on these and other settings, and also around the 'net — are smtpd_recipient_restrictions, smtpd_client_restrictions and smtpd_helo_restrictions, though you'll want to have a look at more than just those within the Postfix documentation.

A server breach can ruin your whole day, too. If you have important data or private data, you might want to consider configuring a DMZ, and keeping your internal network segmented from any servers that are externally accessible. Neither a firewall nor a DMZ is a panacea, but this can make extending a web or mail server breach further into your network more difficult. The most they can access or erase is the breached server.

mhadjar wrote:

I understand its best to put the mail and web servers on the DMZ side of the network. This would be just for the public interface, correct? I can add a thunderbolt-ethernet and have it attached to my internal (private) network for Open Directory authentication? Otherwise, how else do users authenticate with the Mail service?

If that's the case, how does having it on the DMZ vs port forward/NAT on the internal network add security?

A DMZ is a separate hunk of your network, firewalled from the Internet and also firewalled from your LAN.

The goal of a DMZ is to contain a breach, and to make extending the breach more difficult.

Once a server is breached, then it's potentially entirely accessible to the attacker. If you're using the same server where you have your own files and your credit card data or business records or whatever else, then you're going to have a Bad Day.

With a DMZ, the damage is (hopefully, theoretically) isolated to the DMZ and the DMZ host(s), and both inbound and outbound connections to and from the DMZ server(s) are restricted, and network connections from the Internet and from the DMZ hosts into your core network are equally restricted, if not blocked.

Beyond the data on the server, consider that If you make backups on your mail server, then those backups can potentially be deleted by an attacker. If copies of those same backups are automatically pulled from the DMZ host into your internal network, then copies of those backups are likely not accessible to your server. So if the server is breached, you have something to restore.

Do you need a DMZ? No. Do you need a firewall? No. It's another layer of security, so that if a vendor or a user or an administrator makes a mistake — and mistakes happen...

mhadjar wrote:

...but this is a learning experience...

Start reading the Postfix documentation and discussions. Between Postfix and DNS and security and dealing with spam, there's more than a little to learn here, certainly.

The other purpose of this is for any home automation stuff that requires an email address assigned, I'd like to use it for my domain as well. So companies like Nest, Flir, etc can email me regarding software updates, alerts, etc as well.

I often mix locally-hosted mail and domain hosting — both approaches can use a domain that you've registered, too.

I'm not sure if I can restrict checking mail to a VPN connection, if I'm still able to send and receive email from these third-party companies, but that may help with adding a layer of security if it works that way.

Properly SSL'd mail — the TCP ports that were discussed earlier — does keep your traffic secured. Set up properly, your connections are encrypted and the same mail profiles work locally and remotely. Configuring and raising a VPN is feasible certainly — I'd also prefer to use a firewall with a VPN server embedded rather than trying to pass a VPN through NAT, as VPNs and NAT tend to work at cross-purposes — but SSL'd TCP 465 and 993 is just as secure, and it's compatible with how the clients expect to operate. But you can't VPN port 25, and the riffraff will probe port 25 for weak passwords and for open relays and simply for somebody new to send spam to, among other shenanigans.

Also with TCP port 25 — for other mail servers to send you mail or to receive and accept your mail via TCP port 25 — you need to have your host's DNS set up correctly at your ISP and on your local network. Otherwise — again — those other mail servers will drop outbound mail, and may drop inbound mail. (These DNS configuration issues are very common problems when folks that new to running mail servers get started, too.)

SMTP Mail and DNS and SSL/TLS and authentication and security and backups... are all part of running a server. You'll learn a whole lot here, too.

mhadjar wrote:

I delete all public DNS records for my domain except the two mx records that point to MX Logic's service and one host for vpn to connect to.

This does nothing for security — the bots scan for open ports, and scan DNS for MX records separately — and the lack of public DNS makes the mail server harder for you to use, and makes it exceedingly likely that no other mail servers will accept outbound mail from the server barring an authenticated relay.

mhadjar wrote:

On my router, I still have ports opened for SMTP and IMAP, but I set firewall rules to restrict only to the IP's of MX Logic's service. That way, my only threat can come from them directly. And when I VPN, I can use my local DNS for smtp/imap, right?

You'll have to track the IP addresses of all of the mail servers that the ISP uses, and — with the external DNS intentionally misconfigured — the ISP mail servers may not accept inbound mail from your mail server other than via an authenticated relay.

Making the VPN a prerequisite here will also mean that your mobile mail clients will report disconnects when operating off your network, and — due to the lack of DNS — you'll have to hard-code your IP address. Hopefully your router supports "reflecting" outbound network traffic for your public IP address, or otherwise you're going to need to have entries for both your private IP address or private DNS name and your public IP address for your roving mail clients.

I'd not do this. But this is your server.