Apple’s Worldwide Developers Conference returns June 10, 2024

Looks like no one’s replied in a while. To start the conversation again, simply ask a new question.

Hacked?

I have my server set up as a mail server just for our internal emails throughout my department (only 7 email accounts). Today one of the users had over 2300 emails in his inbox, all were bounced back to him saying that he sent it. They were all a typical phishing email, asking the user to click here to update their information for a bank type of scheme. I shut down the mail service, but as soon as I start it back up, the mail.log file starts going crazy. I know very very little about email server type stuff, so I don't really know where to look to find out exaclty what is going on. Here is just a random selection of the mail.log file so you have an idea:

Jan 25 09:37:07 xserve postfix/cleanup[24724]: 3BDBC1A2F3C: message-id=<20070125153530.3BDBC1A2F3C@xserve.edit.oma>
Jan 25 09:37:07 xserve postfix/qmgr[114]: CC6DE1969F3: from=<curt@xserve.edit.oma>, size=1402, nrcpt=1 (queue active)
Jan 25 09:37:07 xserve postfix/pipe[25222]: C3C3718C4CE: to=<curt@xserve.edit.oma>, relay=cyrus, delay=3314, status=sent (xserve.edit.oma)
Jan 25 09:37:07 xserve postfix/qmgr[114]: C3C3718C4CE: removed
Jan 25 09:37:08 xserve postfix/qmgr[114]: 3E8C918C029: removed
Jan 25 09:37:08 xserve postfix/qmgr[114]: 3E128192EB8: removed
Jan 25 09:37:08 xserve postfix/qmgr[114]: CC7671A1E0C: from=<curt@xserve.edit.oma>, size=1405, nrcpt=1 (queue active)
Jan 25 09:37:09 xserve postfix/smtp[15262]: 3DCC9192161: host orngca-01.mgw.rr.com[66.75.160.136] refused to talk to me: 421 #4.4.5 Too many connections to this host.
Jan 25 09:37:09 xserve postfix/qmgr[114]: CC7A518F411: from=<curt@xserve.edit.oma>, size=1397, nrcpt=1 (queue active)
Jan 25 09:37:09 xserve postfix/smtp[20106]: 7A2E9138513: to=<chris.coffin@insightbb.com>, relay=gateway.insightbb.com[74.128.0.19], delay=11175, status=deferred (host gateway.insightbb.com[74.128.0.19] said: 452 Too many recipients received this hour (in reply to RCPT TO command))
Jan 25 09:37:09 xserve postfix/cleanup[24939]: 474961A2F45: message-id=<20070125153707.474961A2F45@xserve.edit.oma>
Jan 25 09:37:09 xserve postfix/cleanup[25254]: 6CFAB1A2F49: message-id=<20070125153709.6CFAB1A2F49@xserve.edit.oma>
Jan 25 09:37:09 xserve postfix/cleanup[25369]: 575D71A2F46: message-id=<20070125153707.575D71A2F46@xserve.edit.oma>
Jan 25 09:37:09 xserve postfix/pipe[25482]: C3C5718A27A: to=<curt@xserve.edit.oma>, relay=cyrus, delay=3632, status=sent (xserve.edit.oma)
Jan 25 09:37:09 xserve postfix/qmgr[114]: CC7EE195660: from=, size=3457, nrcpt=1 (queue active)
Jan 25 09:37:09 xserve postfix/qmgr[114]: 3E91C191519: removed
Jan 25 09:37:09 xserve postfix/pickup[21037]: 61C7E1A2F47: uid=1027 from=<curt>
Jan 25 09:37:09 xserve postfix/cleanup[25032]: 61C7E1A2F47: message-id=<20070125153323.61C7E1A2F47@xserve.edit.oma>
Jan 25 09:37:09 xserve postfix/smtp[18301]: 3E752185856: to=<janet_loeffler@acco.com>, relay=mailgate.acco.com[216.143.30.97], delay=4395, status=bounced (host mailgate.acco.com[216.143.30.97] said: 550 <janet_loeffler@acco.com>: Recipient address rejected: User unknown in relay recipient table (in reply to RCPT TO command))
Jan 25 09:37:09 xserve postfix/qmgr[114]: C3C5718A27A: removed
Jan 25 09:37:09 xserve postfix/qmgr[114]: CC836193B75: from=<curt@xserve.edit.oma>, size=1403, nrcpt=1 (queue active)
Jan 25 09:37:09 xserve postfix/pickup[21037]: CE4151A2F4E: uid=1027 from=<curt>
Jan 25 09:37:10 xserve postfix/cleanup[24724]: CE4151A2F4E: message-id=<20070125153332.CE4151A2F4E@xserve.edit.oma>
Jan 25 09:37:10 xserve postfix/qmgr[114]: 3EA7A19318F: removed
Jan 25 09:37:10 xserve postfix/qmgr[114]: 3DC9318DA40: removed
Jan 25 09:37:10 xserve postfix/smtp[19500]: 3DF511885D4: to=<glen@glentodd.net>, relay=mx1.mailhop.org[63.208.196.176], delay=3992, status=bounced (host mx1.mailhop.org[63.208.196.176] said: 550 Sender verify failed (in reply to MAIL FROM command))
Jan 25 09:37:10 xserve postfix/smtp[11235]: 3EC7C18B9FA: to=<RKCKSYST@aol.com>, relay=mailin-04.mx.aol.com[64.12.138.89], delay=3516, status=bounced (host mailin-04.mx.aol.com[64.12.138.89] said: 550 REQUESTED ACTION NOT TAKEN: DNS FAILURE (in reply to MAIL FROM command))
Jan 25 09:37:10 xserve postfix/cleanup[24939]: 6D3751A2F55: message-id=<20070125153710.6D3751A2F55@xserve.edit.oma>
Jan 25 09:37:10 xserve postfix/qmgr[114]: CC84F18EADE: from=<curt@xserve.edit.oma>, size=1401, nrcpt=1 (queue active)
Jan 25 09:37:10 xserve postfix/qmgr[114]: CC84F18EADE: to=<gregc2@bellsouth.net>, relay=none, delay=3530, status=deferred (delivery temporarily suspended: connect to mx01.mail.bellsouth.net[205.152.58.33]: Connection refused)
Jan 25 09:37:10 xserve postfix/pickup[21037]: 6DFBC1A2F56: uid=1027 from=<curt>
Jan 25 09:37:10 xserve postfix/cleanup[25254]: 6DFBC1A2F56: message-id=<20070125153352.6DFBC1A2F56@xserve.edit.oma>
Jan 25 09:37:10 xserve postfix/smtp[4649]: 7A52216AC5A: to=<dylanjosh@merseymail.com>, relay=mail.merseymail.com[193.110.243.35], delay=6797, status=deferred (host mail.merseymail.com[193.110.243.35] refused to talk to me: 421 argon.connect.org.uk: Too much load; please try again later)
Jan 25 09:37:10 xserve postfix/smtp[20318]: 1B11118DC8E: to=<bradfel@yahoo.com>, relay=d.mx.mail.yahoo.com[216.39.53.2], delay=3282, status=deferred (host d.mx.mail.yahoo.com[216.39.53.2] refused to talk to me: 421 Message from (68.15.230.226) temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html)
Jan 25 09:37:10 xserve postfix/qmgr[114]: 3E752185856: removed
Jan 25 09:37:10 xserve postfix/pickup[21037]: C5D871A2F5D: uid=1027 from=<curt>
Jan 25 09:37:10 xserve postfix/cleanup[25369]: C5D871A2F5D: message-id=<20070125153550.C5D871A2F5D@xserve.edit.oma>
Jan 25 09:37:10 xserve postfix/pipe[25222]: C3CCA185DF2: to=<curt@xserve.edit.oma>, relay=cyrus, delay=4214, status=sent (xserve.edit.oma)
Jan 25 09:37:10 xserve postfix/qmgr[114]: CC8591A07BE: from=, size=3258, nrcpt=1 (queue active)
Jan 25 09:37:11 xserve postfix/smtp[14797]: 3DD101898F3: to=<jeschure@hargray.com>, relay=hargray.com.infoave.mail1.psmtp.com[64.18.4.10], delay=3893, status=sent (250 Thanks)
Jan 25 09:37:11 xserve postfix/qmgr[114]: C3CCA185DF2: removed
Jan 25 09:37:11 xserve postfix/qmgr[114]: 3DD101898F3: removed
Jan 25 09:37:11 xserve postfix/cleanup[25032]: D473A1A2F60: message-id=<20070125153710.D473A1A2F60@xserve.edit.oma>
Jan 25 09:37:11 xserve postfix/smtp[21451]: 3E6F718E940: to=<rentals@carolinabeachrealty.net>, relay=carolinabeachrealty.net[70.87.126.130], delay=3234, status=bounced (host carolinabeachrealty.net[70.87.126.130] said: 550-Verification failed for <curt@xserve.edit.oma> 550-unrouteable mail domain "xserve.edit.oma" 550 Sender verify failed (in reply to RCPT TO command))
Jan 25 09:37:11 xserve postfix/smtp[21331]: 3C6BA190AD5: to=<mbattag2@nycap.rr.com>, relay=clmboh-02.mgw.rr.com[65.24.7.15], delay=2999, status=bounced (host clmboh-02.mgw.rr.com[65.24.7.15] said: 553 #5.1.8 Domain of sender address <curt@xserve.edit.oma> does not exist (in reply to MAIL FROM command))
Jan 25 09:37:11 xserve postfix/pipe[25482]: C3D4517F99D: to=<curt@xserve.edit.oma>, relay=cyrus, delay=4943, status=sent (xserve.edit.oma)
Jan 25 09:37:11 xserve postfix/qmgr[114]: CC87018E8EB: from=<curt@xserve.edit.oma>, size=1402, nrcpt=1 (queue active)
Jan 25 09:37:11 xserve postfix/qmgr[114]: C3D4517F99D: removed
Jan 25 09:37:11 xserve postfix/smtp[20198]: 3DFE018E19B: to=<Kenya@dgrguns.com>, relay=addr-mx01.addr.com[38.113.244.145], delay=3333, status=bounced (host addr-mx01.addr.com[38.113.244.145] said: 553 5.1.8 <curt@xserve.edit.oma>... Domain of sender address curt@xserve.edit.oma does not exist (in reply to MAIL FROM command))
Jan 25 09:37:11 xserve postfix/smtp[19532]: 3EC27185A19: to=<mwland@fuse.net>, relay=mx3.fuse.net[216.68.8.213], delay=4397, status=bounced (host mx3.fuse.net[216.68.8.213] said: 553 xserve.edit.oma does not exist (in reply to end of DATA command))
Jan 25 09:37:11 xserve postfix/smtp[19457]: connect to bellsoputh.net[212.227.34.3]: Connection refused (port 25)
Jan 25 09:37:11 xserve postfix/qmgr[114]: 3EC7C18B9FA: removed
Jan 25 09:37:11 xserve postfix/smtp[19457]: 3EE0018AF5E: to=<PRose1103@bellsoputh.net>, relay=none, delay=3636, status=deferred (connect to bellsoputh.net[212.227.34.3]: Connection refused)
Jan 25 09:37:11 xserve postfix/smtp[20621]: 7770211B171: host desperate.cnchost.com[207.155.253.190] said: 450 <curt@xserve.edit.oma>: Sender address rejected: Domain not found (in reply to RCPT TO command)
Jan 25 09:37:11 xserve postfix/qmgr[114]: CC8861902D1: from=<curt@xserve.edit.oma>, size=1402, nrcpt=1 (queue active)
Jan 25 09:37:12 xserve postfix/cleanup[24724]: E09E31A2F6E: message-id=<20070125153710.E09E31A2F6E@xserve.edit.oma>
Jan 25 09:37:12 xserve postfix/cleanup[25032]: 3FF301A2F71: message-id=<20070125153712.3FF301A2F71@xserve.edit.oma>
Jan 25 09:37:12 xserve postfix/cleanup[25254]: 3FBA01A2F70: message-id=<20070125153712.3FBA01A2F70@xserve.edit.oma>
Jan 25 09:37:12 xserve postfix/qmgr[114]: CC89B19B678: from=, size=3356, nrcpt=1 (queue active)
Jan 25 09:37:12 xserve postfix/cleanup[25369]: 41C211A2F73: message-id=<20070125153712.41C211A2F73@xserve.edit.oma>
Jan 25 09:37:12 xserve postfix/pickup[21037]: 404E21A2F72: uid=1027 from=<curt>
Jan 25 09:37:12 xserve postfix/cleanup[24939]: 404E21A2F72: message-id=<20070125153358.404E21A2F72@xserve.edit.oma>
Jan 25 09:37:12 xserve postfix/qmgr[114]: 42FDBFFE78: from=<curt@xserve.edit.oma>, size=1391, nrcpt=1 (queue active)
Jan 25 09:37:12 xserve postfix/qmgr[114]: 3DF511885D4: removed
Jan 25 09:37:12 xserve postfix/qmgr[114]: 3DFE018E19B: removed
Jan 25 09:37:12 xserve postfix/qmgr[114]: 3E6F718E940: removed
Jan 25 09:37:13 xserve postfix/qmgr[114]: 3EC27185A19: removed
Jan 25 09:37:13 xserve postfix/pipe[25222]: C3D99185D1D: to=<curt@xserve.edit.oma>, relay=cyrus, delay=4224, status=sent (xserve.edit.oma)
Jan 25 09:37:13 xserve postfix/pickup[21037]: 7A22C1A2F7A: uid=1027 from=<curt>
Jan 25 09:37:13 xserve postfix/cleanup[25032]: 7A22C1A2F7A: message-id=<20070125153315.7A22C1A2F7A@xserve.edit.oma>
Jan 25 09:37:13 xserve postfix/smtp[29658]: 3EE7B1884AB: to=<smwhitson@cox.net>, relay=mx.east.cox.net[68.1.17.3], delay=4022, status=bounced (host mx.east.cox.net[68.1.17.3] said: 550 <curt@xserve.edit.oma> sender rejected (in reply to MAIL FROM command))
Jan 25 09:37:13 xserve postfix/qmgr[114]: CC8A61A253B: from=<curt@xserve.edit.oma>, size=1400, nrcpt=1 (queue active)
Jan 25 09:37:13 xserve postfix/qmgr[114]: C3D99185D1D: removed
Jan 25 09:37:13 xserve postfix/pickup[21037]: AEBE21A2F82: uid=1027 from=<curt>
Jan 25 09:37:13 xserve postfix/cleanup[25254]: AEBE21A2F82: message-id=<20070125153351.AEBE21A2F82@xserve.edit.oma>
Jan 25 09:37:13 xserve postfix/qmgr[114]: CC8AC199C39: from=<curt@xserve.edit.oma>, size=1395, nrcpt=1 (queue active)
Jan 25 09:37:13 xserve postfix/cleanup[25369]: B789C1A2F83: message-id=<20070125153713.B789C1A2F83@xserve.edit.oma>

And here is the contents of the email that was being sent:

From: MAILER-DAEMON@xserve.edit.oma (Mail Delivery System)
Date: January 25, 2007 7:58:02 AM CST
To: curt@xserve.edit.oma
Subject: Undelivered Mail Returned to Sender

This is the Postfix program at host xserve.edit.oma.

I'm sorry to have to inform you that your message could not be
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

The Postfix program

<edited forpost@aol.com>: host mailin-01.mx.aol.com[205.188.156.185] said: 550
REQUESTED ACTION NOT TAKEN: DNS FAILURE (in reply to MAIL FROM command)
Reporting-MTA: dns; xserve.edit.oma
X-Postfix-Queue-ID: 103D81570CF
X-Postfix-Sender: rfc822; curt@xserve.edit.oma
Arrival-Date: Thu, 25 Jan 2007 07:15:45 -0600 (CST)

Final-Recipient: rfc822; edited forpost@aol.com
Action: failed
Status: 5.0.0
Diagnostic-Code: X-Postfix; host mailin-01.mx.aol.com[205.188.156.185] said:
550 REQUESTED ACTION NOT TAKEN: DNS FAILURE (in reply to MAIL FROM command)

From: service101@bankofamerica.com <service101@bankofamerica.com>
Date: January 25, 2007 7:15:45 AM CST
To: edited forpost@aol.com
Subject: Bank of America Service Please Update Your Profile - Personal Information Error

Dear Bank of America Customer,

During our regularly scheduled account maintenance and verification procedures,
we have detected a slight error in your account information.
To securely confirm your personal information please click on the link bellow:

http://www.bankofamerica.com/sas/sitekey/profile/step1.htm

Confirm Your Bank of America Account and SiteKey now to enjoy the benefits of
online banking and finance to avoid identity theft and fraudulent activities on
your account.

Note: We will be upgrading our yearly SSL EncryptedServer to prevent fraudulent
activity.

© 2007 Bank of America Corporation. All rights reserved.

If you need more of the log or another log please let me know. So I have a few questions:

1) What exactly is going on? Is my server some type of zombie machine sending out emails?

2) I have port 25 blocked on my firewall - doesn't this mean my computer can't be an open relay?

3) How can I stop it?

Any help would be greatly appreciated.

Quad G5, Mac OS X (10.4.8)

Posted on Jan 25, 2007 9:47 AM

30 replies

Jan 25, 2007 11:33 AM in response to Brent Hilgenkamp

Brent,

You should start with pterobyte's suggestion to run a full virus and malware
scan on the Windows machine.

No endorsements, but you could look here for suggestions.
These tools must be used with care.
I'd also suggest using running chrootkit and/or rkhunter on your
XServe.

-Wayne

Jan 25, 2007 12:27 PM in response to ParentalUnit

I will try all of these suggestions. Is there any log that records what IP or hostname or anything really that can point to where the email originated?

Jan 25, 2007 12:32 PM in response to Brent Hilgenkamp

Brent,

To quote pterobyte:
"If not look at your mail.log. It should be all in there."

-Wayne

David_x

Level 4

3,011 points

Jan 25, 2007 5:48 PM in response to Brent Hilgenkamp

I will try all of these suggestions. Is there any log
that records what IP or hostname or anything really
that can point to where the email originated?

If the outgoing emails are originating from a LAN machine then you will have, in mail.log, a handover from the client to the server. Filter for "connect from". If there is not a hostname for the client then it will be "connect from unknown [ip.add.re.ss]" so you could also do a search on IP address of the client.

I saw another post (other forum?) from you asking about SSH hacking which suggest you suspect the emails originated on the server. This is certainly possible in which case there will not be the connections from the client (with the same sender name as in the emails). I have seen this myself on a home based test server which had open ssh ports and the spam hacker got in via a trivial password for a test user and sent out thousands of spam before I noticed it.

-david

Jan 28, 2007 5:20 PM in response to David_x

Guys,

Hate to spoil the party.. but I've been running a relatively secure server for the past 3 years.. and all of a sudden WHAM!! Someone managed to set up a spurious email account on it (I believe CYRUS / webmail was the exploited weakness) and spewed a whole bunch of junk out on to the internet... all bouncebacks coming back to postmaster told me something was going on.

First I thought it was spoofing.. but then... I noticed the "new" email account!! I have shut down our relatively unused webmail service till we figure out how jerkyboy got in!

OS X.4.8 - ALL the most recent security updates INSTALLED...

Exploit took place sometime around Jan 24th... same time ...

Is there something new out there that we need to figure out?? I don't think this is a random Windoze issue

Jan 28, 2007 5:23 PM in response to Rohin Hattiangadi

er... Just one point... I did leave ssh access open on a wide range of ip's since I have verizon dsl at home.. and I come in over quite a few different Verizon subnets...

STUPID STUPID STUPID... Have obviously shut that down...

Also wanted to point out that there was absolutely NO TRACE of the OUTGOING emails in the mail.log!! First place I checked.

I think this is something new.. has anyone else come under such an attack.

The perpetrator was plugging CanadianPharmacy.com - Hosers! =)

Jan 28, 2007 7:49 PM in response to Rohin Hattiangadi

Hello,

I just came across your post and hope I am not being redundant. But here are a couple of things you may want to try. Have you done a recent port scan of your server (I prefer nmap but the built-in network utility will do fine)? You can download and run the free nessus vulnerability scanner http://www.nessus.org/. Other than that what services do you have running on your server besides mail. You mentioned webmail (www) and SSH. Are there any other services running? Are you running anything like awstat or some php scripts on your website? In the terminal look at the open files with the command "lsof" you can filter it to show files with network connections using "lsof -i". Does netstat show anything unusual? Does the terminal command ps -aux show anything processes running that shouldn't? Is there anything unusual in the http error log? What about in the wtmp logs, which show recent interactive logins (in the terminal type last)? Does your system and secure logs indicate anything unusual failed login attempts followed by a successful attempt? If this activity is still going on you can capture traffic to your server using tcpdump and analyze it using wireshark. You may want to have your firewall log all traffic coming into and out of ports 22, 25 and 80. This will enable you to see what IP addresses are communicating with your server. Hope some of that is helpful. If there is anything I can do I would be glad to help out.

Best of luck!

- Barrett

davidh

Level 4

1,890 points

Jan 28, 2007 7:54 PM in response to Rohin Hattiangadi

Something new ? Sorry to spoil your party but no, not by a longshot.

Securing ssh is important.
http://www.google.com/search?q=securing+ssh

That is most very probably how you got hacked.

That, or any 3rd party software installs, many php-based items have a track-record of serious vulnerabilities.

davidh

Level 4

1,890 points

Jan 28, 2007 8:31 PM in response to davidh

Barrett's covered quite a bit you can do but you may need to reinstall in the end.

In the future, setup ssh key authentication only, allow protocol 2 only (set in sshd_config), explicitly deny root access (sshd_config), and change the port for ssh.

See
http://www.bombich.com/mactips/rsync.html
starting with "Before you start: Security Briefing"

Jan 29, 2007 9:42 AM in response to davidh

David.. no.. no.. let me rephrase...
Just that I thought I had a pretty secure server.. I pretty much shut off ssh by my firewall except for a few select ip ranges... and of course some hacker on verizon dsl network came in through one of these ranges.

I was stupid! Just didn't think that with all the precautions I had already taken that we would be a desirable target.

Mac OS X server was pretty safe in that there were not many hackers out to exploit it.. i.e. now the party is over... It's Unix based popularity is now taking hold and we are now targets.. that's what I meant.

I love the links you sent, plus I would recommend users to use their FIREWALL - Pretty bulletproof to shut down ssh access EXCEPT from the (keep it small) LIMITED ip range from which you will need to access it.

You can stop a lot of poking around in this manner.

ALSO you can use ServerAdmin to turn on the firewall access from this range when you need it, and turn it off when you don't!

Safety first - I learned the hard way... 12 hours of labor to get it all working again

Jan 29, 2007 9:49 AM in response to Barrett Hartman1

Barret,

THANK YOU!!! Believe it or not, had worked on this remotely... Have deleted the suspect account.. shut off all SSH access... checked the logs and processes.... frankly wouldn't know what would be "suspect" (ps-ax) but nothing jumps out.

I think this is shut down, but will of course do a FULL virus scan when I am back at the office later today.

Thanks again....

PowerMac G5, xserve Mac OS X (10.4.8)

Jan 30, 2007 2:55 PM in response to Rohin Hattiangadi

Just an Update :-
I now "believe" that I was hacked via SSh open door.

I noticed that there was activity (outbound) on port 25, HOWEVER these were sent as "camouflaged" dns queries!! i.e. There was absolutely NO unauthorized outbound emails in my mail logs.

Based on recommendations by Barret I started logging ALL firewall activity (Denials and Acceptances) and then grep'd as follows :
cd /var/log
grep ":25 out" ipfw.log

My theory is that someone came in through the SSh door, planted a rootkit, and started spewing spam outbound completely circumventing the standard mail system(?) and it's logs.

Does this sound like sci-fi? As I see multiple outbound DNS TCP port 25 queries to the same server xx.xx.xx.xx etc. - like 15-40 in a row, and then the next server yy.yy.yy.yy.. and on and on!!

Today I just moved over to my clean backup and monitored cpu activity and network outbound activity and thankfully it has all stopped.

WHAT AN ORDEAL!!

I am afraid this is something new that we ALL should be aware of, because this is a hacker sophisticated enough to get into password protected OS X, and send spam in a way where he would not be detected using "normal" mail log monitoring.

Hence will start a new post, and give full details.. just so others in the community are a little better prepared if this happens to them
(See Hacked II)

Jan 30, 2007 3:23 PM in response to Rohin Hattiangadi

update:- some outbound port 25's queries are apparently a normal function of DNS?(!) - like verifying a mail server's address etc.

Still trying to figure out how the spam issued from our server... there is still that phantom account on our server.. so I know it was being used. Question is how? and how is this usage not reflected on mail.log?

The problem is fixed, but I would love to figure out how this was done... and why it was not traceable on the logs?

Jan 30, 2007 5:28 PM in response to Rohin Hattiangadi

Just to continue the discussion on this a bit more and because security is a good (and important) topic.

SSH (TCP port 22) is a fairly frequently scanned port http://isc.sans.org/port.html?port=22 . The most common threat I have encountered are SSH brute force dictionary attacks. These typically attempt to login with various common usernames and passwords. You will see the failed connection attempts in your system log 10.3 and in your secure log 10.3, 10.4. For more information on these type of attacks.

http://www.whitedust.net/article/27/Recent%20SSH%20Brute-Force%20Attacks/
http://www.google.com/search?rls=en&q=sshbruteforce

Some defenses that you can take are moving SSH to listen on a non-standard port. Using RSA keys as part of your authentication. Disabling root login and only using protocol 2. Using a access control list (ACL) and only allowing certain users with strong passwords access to SSH (10.4+ only). Blocking all SSH access to the outside and access it though a VPN (my current solution).

That said in general it is a good idea to know what kind of traffic is entering and exiting your network.

On to cases where a system was compromised via SSH:

On the systems that I have encountered compromised by a SSH brute force attack in the wild they all had a few things in common.

- The system was compromised though a weak password, not a flaw in the OS. (Weak here meaning that it was in the attackers dictionary)
- While the system was obviously compromised through a script the system appeared to be accessed by a person. (there were typos in the Bash history file of the compromised account. While this is not a guarantee that it was a human and not a script it is a fairly good indicator)
- The attacker repeatedly checked to see who was on the system terminal command "w" and what processes were running "ps -ax". (in one case the attacker killed an active process)
- In all but one instance there was no attempt made to cover their tracks (with the exception of hidden directories)
- Each compromise had psybnc installed http://www.psybnc.at/about.html
- Each compromise had several hidden/invisible directories created (by hidden/invisible I mean starting with a "." such as ".s" or the entire directory name of " "). Not all of the locations were the same over all of the compromises. Here is a brief listing of where some were "/tmp", "/var/tmp", "~/.ssh", "~/.sshd" and "~/". This is not a complete list. Just the most common locations.
- One of the compromised systems was setup to be a phishing server. The results were then sent to yet another server.
- There was a list of e-mail addresses to send mail to located in one of the hidden directories. (one server there was a list of just around a million addresses)
- The list of addresses was accompanied by a form letter and a script to mail them out. Usually using sendmail.
- One compromised system looked like the bash shell was replaced with a backdoor. The bash process was connecting to an IRC network.

As for the current case:

- I would double check the /var/log/wtmp logs. You can read them by using the "last" command in the terminal. If the system was compromised over an interactive shell then the logins would show up there as well as the IP where the attack came from. (of note this may not be the IP address of the attacker. The attack could have been tunneled through another compromised system or the IP could have been spoofed.)
- depending on the version of OS X you are running the history of the attack should show up in the system and secure logs. They should look like a bunch of failed login attempts from an IP address that you don't typically connect from. Followed by a successful connection from an IP that you typically don't connect from. You may not have to full history of the attack due to log rotation.
- I would look for hidden directories and an IRC client/server/bouncer
- You may want to make a image of your compromised server so that you can look at it later offline and rebuild the server from a known good backup or from scratch. It is the best / only real way to make sure that nothing got left behind.
- You may want to look at other attack vectors as well until you determine the cause of the compromise.

On a heads up note I have noticed an increase in FTP brute force scans over the past 8 months or so. I have yet to come across a system that was compromised by this but I am interested if anyone has any experience with it other than being scanned.

Hope this was interesting / useful.

Jan 30, 2007 6:48 PM in response to Barrett Hartman1

Barrett,

Thanks again for ALL the good info... As you can probably gather, I essentially just chucked the whole server setup in the toilet, and started over from my 1 week old backup on a nice clean drive.. Not a bad deal.

Yup as you describe there were all sorts of random directories.. user directories (in var/imap/spool) and even after I deleted it, it kept popping back after a few hours. CPU usage was pretty much redlined!

I am pretty certain that the attack came in via ssh, and I also am pretty sure as to which ip (Verizon dsl - same subnet as my home machine) - it showed up in system.log and secure.log

Have shut everything down.. NO ssh... At all!! Will turn it on when I need it and turn it off right after I am done.

Nothing in the logs, but there are missing times, etc. so I am sure evil Hacker man has been editing away. Even though he is a low life crook, he was pretty skilled, so very few traces other than what I assume was a required (?) user account through which to spew the spam.

Still no traces in mail.log, and judging by the fact that there was apparently a LOT of spam sent, I doubt he was manually editing this - it looks like their rootkit or whatever is circumventing apple mail server and affiliated logs.

Details are a little above my pay grade.. but I am learning fast...

Essentially I am working off my clean backup drive (Thank God for that!), and will wipe the affected disk down to the bone and use it as the backup going forward.

Just got an EXCELLENT book called Hacking Exposed (5th Edition) by McAfee (the same anti virus folks..) - Like they say "Know Thine Enemy".

Thanks again for all your help!

Hacked?