SSHD Processes Using 100% CPU

No solution (sorry), but you are not alone. Can be observed here as well sometimes.

Just wanted to ditto this issue. Very peculiar. If I disable remote login, of course the sshd processes (a bunch of them) go away, but as soon as I re-enable it they all come back, chewing up all my Mini's CPU.

Just a thought, if these are Internet facing machines, are you being attacked?

Roger

I'd like to note this as well.

I noticed my computer was quite sluggish over the past couple of days so though I would actually look into it. I noticed SSHD was at 99% cpu usage as well.

I did a tcpdump on port 22 and nothing was coming through.

I did a general tcpdump and there wasn't a lot of traffic. Just general arps and switch traffic.

Any more thoughts on this?

Pete

My Leopard system has this problem. My observation is that it happens if I try to ssh into the machine and enter the wrong password - my ssh hangs, and the sshd on the server side sits and uses 100% CPU until I kill it. It's very annoying, and easily reproducible. Can anyone on this thread verify if they see the same behavior?

I don't see it happening if I try to log in as a random account that doesn't exist, but I do see the problem if I try to log in as root, even though root login is disabled by default. If your ssh port is exposed to the internet, this could explain cpu-eating sshd processes appearing randomly - some would-be attacker may be trying to log in to your machine as root.

I am running OS X Server 10.5.1 on a new X Serve and have this problem. Since this is a server on a public network, it is under constant SSH attack. I can't really kill SSH on a production server since that is how we admin it. It is causing the machine to run slow and hot due to the processor utilization. Everytime I kill that process, it obviously comes back quickly since there are tons of bots out there trying to SSH into it all the time with the wrong password.

Apple: Please fix this problem!

Chiming in - I have the exact same issue on only ONE of my 4 XServe (3 are 10.5.1, 1 is 10.4.10). They are all internet facing and have contiguous IP addresses (ie, when there are BF attacks, they are all attacked at once). My machine is not being brute forced - I can verify this with a peek in the secure.log and netstat. However, whenever I try to SSH in MYSELF, after a few days of uptime on the server, the SSHD process skyrockets to 100% and I am unable to SSH in until I reboot the server (ARD still works though). If I ARD in and kill the process, then the machine drops back down towards idle and no new SSHD is spawned until I try again myself. Cycle repeats.


Anyone else seeing something similar? I find it incredibly strange that I'm only seeing this on one of my 10.5.1 XServe (though it is a different hardware configuration - 3ghz instead of 2.66 and it has a SAS drive, I sincerely hope that this is not due to the hardware difference)

Take proper steps to secure your ssh.

I manage a number of OS X Servers (10.4.x) - either public-facing or still reachable via ssh, and have not had this problem.
Set ssh to use a non-standard port (this is NOT extra security, it cuts away the bulk of the dumb/bot scans/attacks), secure ssh at the firewall (allow only the IP ranges you truly need to let in - takes some work but specific ISPs use specific blocks of IPs), and only allow connections via ssh shared keys.

For changing the port, see
http://www.macosxhints.com/article.php?story=20050707140439980

Choose carefully for the alt. port of course.

For ssh keypair auth, see

http://www.afp548.com/article.php?story=20040816224717742

Thanks for the tips David -
I've already been running a brute force detection script that I put together (scans the secure.log file for > 10 failed logins in 10 minutes, then adds to the ipfw block list & distributes the change to all of my machines). Due to the nature of my users it would be highly inconvenient to run a whitelist only operation.

My problem is that I did not have this problem running the same machine under 10.4 server - it only arose after the upgrade (a clean install). My bruteforce detection operation is running fine too - it's nabbing attacks as they come in, but I get stalled SSHD even when there are no attacks coming in. On a somewhat related note, I also run half of a dozen CentOS4 (RHEL clone) servers with fractions of the horsepower as the XServes, with the same (modified) BFD protection measures for 3-4 years now and haven't ever seen anything like this.

Until the recent issues with the 4-core 3ghz/4gb ram XServe, the worst I've ever seen openssh balk under a brute force is a massive slowdown and huge latency on a 600mhz celeron/256MB ram, while experiencing multiple attacks at once. I've seen no attacks like this on my XServe, yet sshd is hanging completely. So what gives?

Is anyone else still seeing SSH issues even after locking it down?

I see similar CPU spinning occasionally trying to launch Terminal.app. I also see the problem on occasion with sshd.

In both cases I think the following kind of message in system.log appears:

Jan 4 17:51:28 calamari kernel[0]: devfs: ttys000: name slot allocation failed (Errno=17)

Since both Terminal.app and sshd are in the business of dealing with virtual terminals, it seems plausible that what is happening in both cases is a tight loop wrapping a system call which is not expected to fail persistently (but which is in this case).

Or something. 🙂

I have locked down SSH (by blocking the port in the firewall), rebooted the server and then launched terminal.app. It hangs and I have to force quit it.

I then open a hole in the firewall to SSH via only 1 secure IP. Reboot server. Try to SSH in. Hangs and see 100% sshd utilization in activity monitor. Simply quitting or force quitting sshd (or terminal.app) does not fix the problem or allow a log-in.

To answer the possible hardware specific problem, this is a dual 2 Ghz machine, not the 3 Ghz machine listed by another poster.

I seem to remember the problem with 10.5 as well but I am also certain I could SSH in after force quitting sshd with version 10.5 and we can't do that with 10.5.1. Perhaps it was not 10.5.1 but a security update that finally broke it for good since we installed them both at the same time.

FWITW, I also have recently put together a port of sshdfilter into an easy OS X install package. The script scans incoming ssh connections and detects brute force attacks, then blocks them. It's helpful if you are in a position where due to your users, you must allow for arbitrary IP's to access on a standard port. The port/installer is still in beta testing, but seems pretty stable and would appreciate any comments if anyone is looking for such a script: http://projects.seas.columbia.edu/sshdfilter/.

No good solutions from me, but I am another sufferer on a mac mini. First it was with sshd, then Terminal.app went south.

It pains me to say it, but the "windows treatment" aka reboot cleared it up, for now. Not a useful solution for those running servers though.

Hope apple fixes this one soon.

New symptom:
Left an ssh session open last night when I went to sleep. Woke up, used it just fine. With it open, attempted to open another session to the same machine: sshd goes up to 99%, but the original session continues to function fine. There are 0 lines in my secure.log in betweeen when I opened the session last night and when I tried logging in this morning.

Jan 8 11:07:45 projects com.apple.SecurityServer[35]: checkpw() succeeded, creating credential for user jsb2125
Jan 8 11:07:45 projects com.apple.SecurityServer[35]: checkpw() succeeded, creating shared credential for user jsb2125

But that's it - it never finishes opening my session.

We are having exact same issue, this is very infuriating, a UNIX box without SSH is like a car without a windscreen.

Two users at our site have experienced different facets of what appears to be the same issue. One user has experienced problems opening multiple Terminal.app windows, in which the system appears to reach a point where it can no longer allocate virtual ttys. What initially causes this state is unknown, but once the system reaches this point, all further attempts to allocate new terminal sessions seems to fail.
The second instance of this was experienced while trying to execute remote SSH sessions. Once again, the remote server system seems to have gotten in a state where it could no longer allocate ttys. We saw a very reproducible condition where one user could login, but any further attempts to login via SSH caused the client process to hang, and the server process to spin with 100% CPU time. Manually killing and restarting the SSH processes was only a workaround to the high CPU utilization, but didn't seem to clear up the tty allocation problem.

Under both cases the same error messages were being printed to the 'dmesg' kernel ring buffer.