I'd not initially suspect a trojan, unless you've been downloading and installing random stuff onto your server.
I'd assume a weak password, or a mis-configured or a down-revision (and vulnerable) web-facing service or web tool. Probes for these sorts of errors and weaknesses are quite common.
Shut down your network, and shut down your mail server.
Get a backup of your disks, on the off chance that there might be attempts to delete your data, or errors made while attempting to clean up the attack.
Reset all of the passwords. All of them. Certificates, too, if you have SSH active.
Then start digging through your logs, and see what's going on here.
Get rid of any files that the attackers have loaded into the web directories.
Get rid of any users that have been added by the attackers.
Make sure you don't have other breached systems, too; either "trusted" systems that are using your OS X Server box to route spam, or other boxes that have been compromised from a breached OS X Server box.
If your attackers have left one or more backdoors, then you can have the nasty choice of finding and disabling each of those that might be around, or reinstalling from distros. Once breached, your system is not trustworthy. (There are all sorts of obscure spots that a backdoor can be hidden away for later use, unfortunately.)
Here's a general write-up on what can be involved in decontaminating a server.
There's no easy way out of a security breach, unfortunately. It's a slog.
