I guess we have been hacked and while I sort it out I want to turn off outgoing mail while still being able to receive.
Anybody know the terminal command to do this?
I also read of the recommendation to block outgoing port 25 and use 587. Anybody know how to block 25 and open 587?
Thanks!
Mountain Lion Server.
Mac mini, OS X Server, null
I'm assuming you have a NAT'd network here, as is typical. If so, then block any inbound and any outbound network connections to port TCP 25 at your network firewall; at the box that's performing NAT, and port forwarding. How you configure that depends on your particular network firewall — it's not something that's local to the now-not-trusted OS X Server box, this block is performed at whatever box is at the edge of your network. Most firewalls can block in-bound traffic, and mid-grade and better firewalls can also block outbound traffic as well.
TCP 587 is the submission port. That's how your clients can access remote servers. Depending on the breach, it could be that port that's being used to send spam, or it might be TCP port 25 and your mail server.
Without a look at the server logs, there's no way to know what the path might be, assuming the breach is isolated to mail and not a more widespread breach.
If you want to try establishing this firewall block at your server — and without knowing the extent of the breach, it's impossible to say whether this will work, or whether your nemesis has sufficient access to just revert anything you might try — you can try using the host firewall. Details here and here, among others.
It's common to have all TCP port 25 access inbound to any system other than your mail server blocked, and to also block any outbound TCP port 25 access that's not from your mail server.
I was hoping you'd see my question MrHoffman, since you always seem to give the most in depth help.
I guess I worry about your saying "now-not-trusted OS X Server box" because we have what I believe is called front facing servers. Two ethernet interfaces, one with the external public IP, and one with the private internal network IP. The reason is that before I was ever involved in setting up the internet stuff, my boss had a guy come in and he set up a server (blue box called a qube), and he configured it that way. So when Apple touted how easy it was to run a server, he bought a mini with SL server. So I just copied the settings from the blue box to the SL server. He likes to have everything in house, so he just throws me in the water to see if I can swim.
So when we got ML server, I thought about NATing, but it seemed like a hassle and just easier to do the settings the same. VPN and ARD seem a snap to use by just connecting to the IP of the server I need. We also have a Yosemite server set up the same way. If this is not as secure as a NAT, is there anything I can add or change to make it any better? Or is that what the links you gave were for?
I have become fairly good at interpreting logs, and I couldn't find anything that indicated any breach or how it got in, but that doesn't mean it wasn't there. Sometimes, there is so much activity in the logs, that they are all full with only the last hour or so of stuff. Is there a way to read any further back than what I can see in console? Is there some deep log archive somewhere? I was reading that if there is a bot on the network, then what it's doing may never show in the logs. That's where I read the tip to block port 25.
Just few other questions:
Since I'm not on a NAT, are there terminal commands that can block port 25 and open 587?
Is there a command to disable outgoing mail while leaving incoming running?
Is there any way to actually "see" the incoming and outgoing traffic and what it's doing other than looking at what the logs show?
Rambling reply...
I'm aware that you're asking me for the command to disable outbound mail. In this case, there's no reliable way to do that. Shut down the server. That is how you deal with these problems. This has unfortunately already been going far too long. Hacked? I'd kill the whole server. Everything. Immediately. The longer these cases drag on, the worse the messes tend to get, and the more blacklists your server gets onto. Once shut down, I'd isolate the box, image it, and then start to investigate the damage — that's not a quick process, either — from another box using Target Disk Mode or swapped disks.
If your mail server box is hooked directly to the Internet and with no intervening firewall box that you can control, then I'd just shut down mail entirely, and troubleshoot and clean up the mess. Quite possibly the whole box, as described. To allow arriving mail to continue, redirect your DNS MX mail record to a different server using your public DNS, either with that server operating as a lower-priority relay server and caching the mail there or — probably better — replicate your accounts with wholly new passwords and host your mail offsite temporarily.
It's common to assume that everything about this breached server is now not trustworthy, too. Not until the breach has been identified and isolated, and — if the box got rooted, as can happen — the entire data contents have probably been exfiltrated, and the programs and settings are not trustworthy, and you're headed for a nuke-and-pave / wipe-and-reinstall sequence. Backdoors and changed passwords and new accounts can and do happen. (Some background on hacked servers.)
There's no easy and no quick way to deal with this, and one of the nasty things with a not-DMZ'd public-facing and rooted server box is that your whole internal network and Wi-Fi passwords and the rest may well all also be toast. How far the attackers got, that might be clear from the logs, or it might well just be safest to assume the worst. That is, nuke-and-pave the box, restoring from either pre-breach backups — if the attacker hasn't hosed those — or from distributions. Alternatively, it's also possible that some box elsewhere on your internal network was compromised, and it's that box that's spewing out spam, either through your mail server or directly.
Without details and without investigation, the extent of what happened is not known.
pf — links referenced in an earlier reply — is the path for shutting off network connections, but — if the attackers got further than this box — then some random network printer — yes, some network-connected printer, I'm not kidding here — can be sending out spam. This is part of why having a firewall — which is not a panacea — can help. Here, your firewall — your OS X Server box — has been breached. But if you trust it, shut off both inbound and outbound TCP port 25 traffic.
As for NAT, I use firewalls with embedded VPN servers. FWIW, NAT itself is not security; it's the port-forwarding settings and the ability to isolate and restrict unnecessary traffic through to the public internet. I'd use a firewall — again, not a panacea — with even a network using public addresses.
As for the logs, the locations on those vary. Console.app can get you a view into the basics, but beyond that you're using the command line. In Mountain Lion Server, look in /var/log and /library/server/Logs and /Library/Logs and a few other paths. Mail.log is at /var/log/Mail.log, but the various logs have moved around in recent OS X Server releases. To find logs, you can control-click on the log name in the left column in Console.app (select Show Log List, if necessary) and choose Reveal In Finder from the pop-up, or you can use that path to locate and read the logs via the command line and Terminal.app.
FWIW, ZyXEL ZYWALL USG is a local favorite and a very capable box with a consistent interface, DMZ and VPN capabilities, but those products are not "introductory" products, They most definitely do assume familiarity with IP terms and concepts and mechanisms. Other than having purchased various gear from ZyXEL, I have no connections with them.
Not sure we were hacked, it was just an assumption. We got on the CBL blacklist. I changed passwords and checked/fixed web space permissions.
It had been over two days since they had detected any activity, so I contacted them to see if this meant anything. They wrote back and said that's generally enough and cleared us from the list without me asking.
Really wondering about this one: How can a printer send spam, and how can you tell if it is? Over the weekend I was checking the devices, and there was one device on the wifi network when there was nobody there. It turned out to be a brother printer at someone's desk. It has a web admin ability and I looked in there and there was nothing in the email/smtp settings.
Trying to get everyone to use imap, which unless I'm more of an amateur than I thought, means we wouldn't need port 25. So is there a terminal command for opening and closing ports?
Also, I thought that while I sort this out, if I could turn off outgoing, people could still get mail and they could use another outgoing server. Thought it would also be a good way to study the logs if I turned it on intermittently while everyone is using another outgoing to see if anything actually goes out.
As for the logs, in ML, I have found all those locations, but it seems to be all that is available in console. Are there no more "deeper" log storage or archives than what's available in these locations?
Thanks for your help!
Another rambling reply...
if you're looking at the contents of the directories from Terminal.app and the command line, then either the logs were disabled and/or have been rotated and deleted, or you've a potentially much larger security problem than a mail server being used for relay spam or spam secondary some compromised account credentials.
With the normal log file rotations, the older logs will have been compressed (usually .gz is appended to the filename to indicate gzip compression), so you'll have to decompress those before viewing, or use the gzcat or zcat (for lz compression) tool.
The contents of and location of the logs themselves are controlled by the particular server settings; show fullstatus for details. Postfix has gonzo capabilities; you can copy outgoing mail, for instance. (See this thread for the postconf command you can use to change the settings, then reload the server.) I've not had the need to increase logging levels to trace messages within the Postfix logs, but that's possible to do. (This ancient article shows the general sequence.)
Correct: IMAP does not use port 25. IMAP uses TCP 143. Secure IMAP (IMAP SSL, etc) uses TCP 993. Some mail clients will use SMTP as the submission port and that's generally not a good idea for various reasons — it's increasingly common to block outbound TCP 25, too. Using the authenticated SMTP submission port TCP port 587 is widely preferable. I would allow TCP port 25 only for mail-server-to-mail-server SMTP traffic, and not for mail clients. TCP 587 for submissions, not 25. TCP 993 for IMAP, and not 143.
Network-connected printers can send spam because they — like most devices these days — contain firmware, some printers have bugs in their firmware, the firmware was not kept current, and somebody was convinced — phished — to print (for instance) a PDF, and the PDF then exercised the hole. Once you control the firmware, you now have a network-connected device on a protected network. Next step is usually an outbound connection to a remote control system, which allows the attackers to access your network via your printer. The printer can then do... whatever. Spam included. Rather than and more common than infested printers, compromised clients can be sending out spam, too — various malware will either send using the local path or might have a full mail server running on the infested client box. This is why I prefer to block outbound TCP 25 connections from everything other than the actual and authorized and intended mail server. I'd be tempted to log all output TCP 25 attempts too, just as it might point to some box I need to look at.
I'd get a firewall, and I'd look to get the mail server into a DMZ; keep it separate from the rest of your network, as it's going to be somewhat reachable from outside. Get your server and your devices and the rest to current versions. Check the mail server logs and find out what's going on. Longer passwords or migrate to certificates. Expunge all use of telnet and ftp. Use ssh and VPNs for remote access. All of the usual recommendations.
As usual, you are a fountain of info.
I found all the logs with .gz are readable in console. So are you saying that all the logs in these places are all there is, no deeper archive somewhere?
I assume if we get the firewall you mention, that is where we would open and close the ports. But until we do, do you know the commands I would need to close 25 and open 587 on the ML server? Also maybe a way to specifically log port 25 as you mention?
As long as I have you here, wondering if you know about something else. We are using woocommerce on Yosemite server and IPN notifications are failing at paypal. The tech at WC said that this is most likely the problem:
Looks like your Apache configuration are SSLVerifyClient set to required which would cause it to act like what you are describing. You'd need to set it to none: http://httpd.apache.org/docs/current/mod/mod_ssl.html#SSLVerifyClient
I tried manually adding "SSLVerifyClient none" to the Apache conf file and it didn't do anything. Any idea what I may have missed?
Thanks!
scottl31 wrote:
I found all the logs with .gz are readable in console. So are you saying that all the logs in these places are all there is, no deeper archive somewhere?
Beyond the specified logs and your backups of same, if there were a deeper archive, I'd have pointed to it.
I assume if we get the firewall you mention, that is where we would open and close the ports. But until we do, do you know the commands I would need to close 25 and open 587 on the ML server? Also maybe a way to specifically log port 25 as you mention?
Some external firewall boxes can log activity. The firewall devices I've referenced earlier do have some logging capabilities.
See some of the previous links for the host-based firewall command syntax and introduction. Disabling TCP port 25 on a mail server will hose SMTP traffic. It's the mail clients that should be migrated from TCP port 25 to TCP port 587, as well as over to SSL ports for POP or IMAP.
I haven't looked for a way to log SMTP beyond the provided logs, though it's generally feasible to configure a Unix server to use syslog or analogous to transmit the log contents to a remote log server of your choice. You're probably not interested in running a dedicated log server, though.
As long as I have you here, wondering if you know about something else. We are using woocommerce on Yosemite server and IPN notifications are failing at paypal.
Separate questions are best asked separately, as my walnut-sized-brain tends to get pretty tangled — as you're probably about to realize here, as you read onward — when the question and the configuration changes (dramatically) in mid-forum-discussion.
You have money on this server or somewhere in this configuration? OK.
That changes a few things. In particular, you're now a much bigger target than a random mail server.
Accordingly, I'd strongly recommend a full-on audit of this system and everything else that was in range of the potential breach — the US National Security Administration (NSA) has some configuration and security recommendations for various operating systems — and I'd likely perform a preemptive ground-up rebuild, and I'd get a firewall and a DMZ configured, I'd minimally upgrade to TLSv1.0 and current-generation crypto everywhere, and lock out SSLv3 and earlier on all of your web services — Mozilla has a pretty good list for crypto, FWIW — and I'd look to get everything to Yosemite Server.
I'd also strongly suggest either some security training or time to really start learning server management and security and/or some formal help, and likely a security review. In general, you're probably going to want to outsource some of your risk. (There are requirements and recommendations beyond what I've previously mentioned in this thread, too. Things that I'd do or configure, and audit and review. This work usually gets rather detailed.)
Looks like your Apache configuration are SSLVerifyClient set to required which would cause it to act like what you are describing. You'd need to set it to none: http://httpd.apache.org/docs/current/mod/mod_ssl.html#SSLVerifyClient
No sé. I'd need more details on that and some time to research that setting; disabling certificate verifications can lead to Bad Things. I'd also need some time to research the current Paypal requirements, as I've not looked at those in a while.
I tried manually adding "SSLVerifyClient none" to the Apache conf file and it didn't do anything. Any idea what I may have missed?
No sé. Guess: maybe the wrong file location or the wrong file name? In recent Server.app releases, the Apache "httpd.conf" configuration file is /library/server/web/config/apache2/httpd_server_app.conf
Locally (and I've not further checked the device-management references shown below) there's no specification in the main Apache configuration files, though :
$ grep -R -i SSLVerifyClient /library/server/web/config/apache2/
/library/server/web/config/apache2//httpd_devicemanagement_ssl.conf: SSLVerifyClient optional
/library/server/web/config/apache2//httpd_devicemanagement_ssl.conf: SSLVerifyClient require
$
FWIW and beyond any applicable Paypal requirements, you're right on the edge of PCI compliance here, too.
Thanks for all the insight. You've given me a lot to work with.
Sorry about asking a different question in the thread. I did ask this in it's own thread and got no replies. Since you have more knowledge than most people on here, I was sort of trying to take advantage of that.
Anyway, the ports/outgoing issue is on ML server. This apache thing is on Yosemite. There's no money or credit card entry on the site, all transactions are on paypal. That's why we don't have an SSL CA. Is it a good matter of course to have a CA on the server anyway? I have read a lot in these forums that they can be a huge pain to deal with or even get working on OSX server.
I was aware that the config file was httpd_server_app.conf, but that other file you mention does have the SSLVerifyClient entries in them as you note. Is it OK to edit such files directly? If so, then I guess I'd simply replace both optional and required to none.
scottl31 wrote:
Anyway, the ports/outgoing issue is on ML server. This apache thing is on Yosemite. There's no money or credit card entry on the site, all transactions are on paypal.
First, there's no DMZ, and apparently open access within the network. That means that a mail server breach can have gotten further. Second, there's all sorts of interesting reading on a mail server, whether it's actual credentials or just fodder for spearphishing folks. You've quite possibly got credentials for Paypal around somewhere, and there can be credentials and access to various other internal and external servers and clients involved — it's rare for folks to by scrupulous with credentials, so ransacking a mail server can be... "entertaining."
Breaches are — as I've linked earlier — not very much "fun" to deal with, and the best path usually involves mapping out where you really want to be, and nuking and paving, and rebuilding, and reconfiguring the network and the servers to get there.
That's why we don't have an SSL CA. Is it a good matter of course to have a CA on the server anyway? I have read a lot in these forums that they can be a huge pain to deal with or even get working on OSX server.
I'm mildly surprised Paypal allows a redirect to their forms from a non-encrypted page. Yes, get a commercial cert. Yes, use SSL/TLS. (I'd be cautious about unencrypted pages, as the folks connecting to your site without the benefit of HTTPS and SSL/TLS may not be getting to your actual Paypal link. Some entities routinely inject ads and JavaScript and other junk into HTTP streams, so it's not necessarily really clear who your customers are actually paying, after all.)
Again: you're running ecommerce. You're a bigger target.
You'll probably want to configure the box in a DMZ, with a public static IP name and host. Public static IP means an easier time with mail and DNS and SSL/TLS certificates.
I was aware that the config file was httpd_server_app.conf, but that other file you mention does have the SSLVerifyClient entries in them as you note. Is it OK to edit such files directly? If so, then I guess I'd simply replace both optional and required to none.
Those two grep'd files are related to mobile device management, and AFAIK should not apply here. I'd leave those alone. (I don't know that the SSLVerifyClient setting is even applicable here. Not without some digging around to determine the details of whatever issues were encountered, too.)
I've edited the Apache configuration files directly in various cases, but doing that there've been occasional problems reported over the years, and it's also possible to make typos or other errors in the files. Minimal alterations to the files are preferable, and the use of an include file where that's available. The Apache file usually does have provisions for loading commands from an external include file; see the <IfDefine WEBSERVICE_ON> stuff for the include.
How do I disable outgoing mail?
