Has your server security been breached? If there are users that have been created on your system, or if the outbound mail is otherwise originating from your server such as a breached web server, then the server has been compromised and the vulnerabilities will need to be addressed, and OS X Server either reinstalled or the issues otherwise resolved.
If you're on the receiving end of a spam run that forged your domain outbound, there's not much you can do other than using your email filters. Those might be bounces from misconfigured mail servers, or the messages you are receiving might be what the spammer is actually sending; the bounces might be the spam.
More details will be required.
There are two potential issues here. One is that the mail bouncing back to you has forged headers with your email address listed as the From or Reply-To address. This is a common problem and is not a result of your server having been compromised. Make sure spam filtering is enabled in the mail server settings in Server Admin to reduce the volume of bad messages in that case.
The second potential issue is that a miscreant has run a password cracker on your mail server and found valid user names and passwords, allowing them to use your mail server to send spam. Make sure plain text passwords are disabled in the mail server settings in Server Admin and change all of the passwords for every user on the server. Also make sure that the SMTP server requires a user name and password for authentication before email can be sent.
I also recommend that you use the firewall to restrict access to critical ports like 22 (SSH) and 5900 (Remote Desktop) to just the address blocks you connect from and deny everything else. Also disable Telnet. Miscreants are constatly scanning the Internet looking for open ports they can use to gain access and crack a server. If your server has been compromised at the root level you will have to re-initialize the hard drive and re-install OS X, as that's the only sure way to eliminate a root-level compromise.
Thank you all.
Sorry for being unprecise. I am basically receiving a lot of messages from MAILER-DAMONS and similar telling me that regretfully they cannot deliver the mail from email@example.com. The thing is that the xxx´s are not mail accounts or users on my server, but fake accounts using my domain.
I do not think my server is compromised and anyone has broken into it and used the server management to create accounts and similar. At least I cannot detect anything like that. My basic problem is that I have a daily stream from different mail administrators with messages of undelivereded messages.
I have asked my server to bounce junk mail, I have moved the junk mail filter to 6 and I have created rules on my mail account (mail system administrator), but still I receive quite a lot of them. Can I somehow tell my mail server only to accept mails to the accounts on the server, and bounce or ditch all other incoming mails to fake accounts using my domain?
Regarding protecting my server better, advice well taken. I believe I have a good firewall in my router, and I will reduce the port forwarding to what is strictly necessarry. I have heard about special security services that one can ask to simulate an attack on the server to just look for weaknesses in the defense. Any tip with regards to that?
Would appreciate more comments.
In that case you're probably just getting blowback from forged headers containing your domain name. Disable the mail setting in Server Admin that redirects mail for unknown users to the admin account and that will bounce them back to the sender.
There are various web sites around that will scan your server for open ports, but you don't need that if you know how your firewall is configured. I recommend allowing traffic from the WAN (Internet side) only to those ports you have services on, like http, mail, etc., restrict access to critical ports like SSH, Remote Desktop, etc., and deny all traffic and protocols from the WAN to everything else.
I would like to implement what you recommend. However, I cannot find "disable the mail settings in Server Admin that redirects mail for unknown uses". I can only find the possibilty to disable messages of undeliverable mail. Is that the one you are thinking of? If not, can you give me a more detailed description?
the ideal would be to block mails to accounts not on the server. Messages telling about undeliverable mails may be very important once in a while.