someone have hijacked my domain

In normal circumstances, your email server would not accept email for email addresses that do not exist, or are you saying they are using email accounts that exist on your server?

Has your server security been breached? If there are users that have been created on your system, or if the outbound mail is otherwise originating from your server such as a breached web server, then the server has been compromised and the vulnerabilities will need to be addressed, and OS X Server either reinstalled or the issues otherwise resolved.

If you're on the receiving end of a spam run that forged your domain outbound, there's not much you can do other than using your email filters. Those might be bounces from misconfigured mail servers, or the messages you are receiving might be what the spammer is actually sending; the bounces might be the spam.

More details will be required.

There are two potential issues here. One is that the mail bouncing back to you has forged headers with your email address listed as the From or Reply-To address. This is a common problem and is not a result of your server having been compromised. Make sure spam filtering is enabled in the mail server settings in Server Admin to reduce the volume of bad messages in that case.

The second potential issue is that a miscreant has run a password cracker on your mail server and found valid user names and passwords, allowing them to use your mail server to send spam. Make sure plain text passwords are disabled in the mail server settings in Server Admin and change all of the passwords for every user on the server. Also make sure that the SMTP server requires a user name and password for authentication before email can be sent.

I also recommend that you use the firewall to restrict access to critical ports like 22 (SSH) and 5900 (Remote Desktop) to just the address blocks you connect from and deny everything else. Also disable Telnet. Miscreants are constatly scanning the Internet looking for open ports they can use to gain access and crack a server. If your server has been compromised at the root level you will have to re-initialize the hard drive and re-install OS X, as that's the only sure way to eliminate a root-level compromise.

In that case you're probably just getting blowback from forged headers containing your domain name. Disable the mail setting in Server Admin that redirects mail for unknown users to the admin account and that will bounce them back to the sender.

There are various web sites around that will scan your server for open ports, but you don't need that if you know how your firewall is configured. I recommend allowing traffic from the WAN (Internet side) only to those ports you have services on, like http, mail, etc., restrict access to critical ports like SSH, Remote Desktop, etc., and deny all traffic and protocols from the WAN to everything else.

Yes, the undeliverable mail option is the one I was referring to. If that's enabled you'll get a lot of spam blowback as a result of forged headers using your domain name with bogus user names. You'll have to decide which is more important to you.