Check your mail server logs, and see if there's anything relevent being logged there when the messages are in-bound.
Also check that your MX record is what you expect, and that its forward and reverse translations match.
To check that, launch Terminal.app from Applications > Utilities and enter the following diagnostic commands. (Stuff that can and will vary for your specific setup is shown in italics, stuff in the commands that doesn't vary uses plain text.)
dig @126.96.36.199 MX +short example.com
That asks Google DNS server at IP address 188.8.131.52 to show your MX record for your mail server, and the output shown assumes that your public mail server name is mail.example.net host.
Now get the IP address...
dig @184.108.40.206 +short mail.example.net
That's your public IP address for your mail server.
Now get the name associated with the IP address that the previous dig command returned...
dig @220.127.116.11 +short 10.20.30.40
Use the IP address that was returned by the previous dig command. If you don't get back your domain name for your MX server, then your public DNS is misconfigured.
I'm also assuming you have just one NIC connected on that Xserve system. If you have more than one NIC connected, there can be or are extra steps, and you'll need to have IP routing correct.