This may be an obvious question, but I wanted to ask before I began on setting things up.
While we currently run iCal & Address Book on our OSX server, I wanted to set up the mail server as well for our small office (5 people). We currently use our ISP's mail service, collecting mail from each workstation individually using POP3. It seems sensible to continue to use the ISP's service to relay mail to our server by SMTP, so that we avoid any problems if its not running all the time. I noticed that the mail server has the facility to relay outgoing mail through an ISP's server, but presumably its also possible to collect mail from the ISP's server as well?
If so, does anyone know is this fairly straightforward and is there anything I should be aware of?
Any advice much appreciated?
In no particular order....
If you use relay for outbound mail from your server (which would be more common if you don't have static IP, or don't have proper external DNS configured for your server), then you can either have your inbound mail land on the ISP servers inbound (and pick up mail as you have), or you (assuming static IP) can have inbound mail land directly on your mail server.
Where the inbound mail goes depends on the DNS MX record for the domain, and whether that mail server is configured to accept that arriving mail.
Whether outbound mail is received by remote servers depends on whether you're using authenticated relay (as you're discussing) and whether you have valid forward and reverse DNS for your mail server (if you're sending mail directly). If your DNS is messed up, receiving mail servers will often interprete your mail server as a spam engine and drop the messages, and various servers won't even bother sending to a mail server with incorrect DNS.
I'd use IMAP and not POP.
If you run your own mail server, you own configuring and maintaining and running the anti-spam defenses and related hassles.
Thanks for that response MrHoffman
In answer to some of your questions, I assumed we'd need a static IP address as I wanted to set up a VPN as well, and our ISP can arrange to send and recieve mail to it via SMTP. I had assumed that we would use the DNS server running on our LAN to handle all internal private addresses, but rely on the ISP's DNS to deal with public addresses. I can see that getting the two to communicate properly is the key - does that sound possible?
Was planning on using IMAP to help manage mail more effectively, which is the main reason for running our own mail server.
The easiest way for inbound mail is to have static IP on your firewall (or your server, if it's exposed), and set up the ISP as a lower-priority (higher-numbered) MX and your server as the higher-priority (lower-numbered) SMTP server. Then mail is delivered directly. If you're on a static IP, then proper external DNS means you can send mail directly outbound (and more importantly, have it received and accepted). This is a full-on local mail server.
If you go this way and deploy an externally-accessible mail server, then you'll need to lock down against forwarding and the plethora of attacks that are launched against mail servers, as well as dealing with the spam and related messes.
My preference with VPNs is to use a firewall (firewall-gateway-router-etc) that implements a VPN server within the box. Then you're not messing with NAT passthrough, and you can also maintain inbound VPN access even when the OS X Server box is offline.
If you're running a private NAT'd network, OS X Server requires valid local DNS implemented on your LAN, or things tend to get wonky. ISP DNS won't work for this, if there's NAT in play.