Are you looking for help setting up the Mail.app client with your new mail server?
Or getting the mail server connected, configured and working?
If it's the client and if you can't receive mail messages (messages that were sent locally within the mail server), then the POP or IMAP path in the mail client configuration doesn't match what the server is required; try the Connection Doctor tool in Mail.app and see what it reports. You'll need to match that with how the mail server is configured.
If this is a problem with receiving mail on the mail server (from remote), then that's completely dependent on correct forward and reverse DNS for the mail server, and matching the MX record for the domain. Remote mail servers variously won't send to mail servers with misconfigured DNS, and increasingly often won't accept mail from mail servers with misconfigured DNS. (Mismatched DNS is typical of spam engines.)
I'm here going to assume you have a network gateway device that is performing NAT, and that you have public static IP. Things get more complex with dynamic IP.
Internet SMTP Mail is fundamentally based on correctly-configured public-facing DNS. For other SMTP servers to accept messages to and from your mail server (and to not assume that it is a spam engine), both forward (host or domain name to IP) and reverse (IP address to name) translations must be correct.
The other way around the need for correct is what's called an authenticated relay, and you'll need to establish that up with an email provider, and not all email providers allow relays. Basically the provider becomes your public connection point, and all mail is "tunneled" through that provider, and your provider has correct DNS.
In-bound HTTP web services are rather more tolerant of DNS, and only depend on valid publuc forward (name to address) translations.