SSHD Processes Using 100% CPU

Same problem here, and nope, 10.5.3 doesn't seem to help. Odd thing is that we hadn't seen this issue for the first ~3 months we've had this server running. At the time, I believe we were running 10.5.1. Then it appeared one morning (couple weeks ago I think) when someone was trying to ssh in (server had probably been running for over a month straight). Things kept working (didn't notice that 20-40% of an 8 core server was tied up). When I finally noticed it, I found this thread, installed 10.5.3 and rebooted. That was a few days ago, and now we're back in the same state. So, I expect now it's going to keep recurring with our current usage pattern. 😟

Hard to speculate what's different. We have a couple more people ssh'ing in than a month ago. Started using Open Directory and migrating some accounts over.

I've just noticed this issue on my Mac OS X 10.5.3 Leopard Server. Same exact symptoms as everyone is describing only I'm on a quad-core Mac Pro.

I tried the SACL fix but it had no effect….

Will read on and hope for a fix. Very annoying…. This wastes a lot of electricity!

Ditto, Craig. My site is running on 10.5.3 and I've not noticed this issue before. We were running 10.5.2 on the same hardware, no problems. I'm certain I've made no changes that would be an obvious trigger for something like this, but clearly the issue is here now.

I do a bit more SSH'ing these days, so maybe that's it…? Hard to tell.

Anyway, just wanted to chime in that I'm experiencing this issue on a fully-patched Mac OS X 10.5.3 Leopard Server machine, running on a quad-core Mac Pro. Thankfully we have more than enough horsepower to handle the extra load, but the this is very frustrating. Our quad-core Mac Pro is essentially a tri-core machine now, one die completely occupied by sshd. Further, this wastes a lot of electricity and that is both expensive and horribly ecologically unfriendly. 😟

I sure do hope Apple fixes this soon.

Just another work-around: A colleague of mine found a possibility to minimize the negative effect of the hanging SSH daemons to almost zero.

He has just sent them a SIGSTOP using the "kill" command.

kill -SIGSTOP <process-ID of_the_hangingSSHD>

This will end the CPU consumption by the hanging process, and additionally, it doesn't seem to trigger new hanging SSH processes because the broken PTY resources that the hanging job uses are still "in use". So we don't run into the "zombie problem" (see my post http://discussions.apple.com/message.jspa?messageID=7291648#7291648) which we would get when sending a SIGTERM or a SIGKILL to a hanging process.

If I had the time, I would write a little shell script that checks for hanging SSHDs (detect 100% CPU usage for some time) and sends them a SIGSTOP automatically. I think that running such a script from time to time (via CRON) in the background would be a work-around until Apple fix this annoying bug.

Message was edited by: Steffen M.

Stefan! That is so simple it just happens to work!

I can verify that this resolves the CPU-hogging state of my Mac Pro 10.5.3 server. It's still a nuisance, of course, but it does at least let the machine rest a bit.

Thanks for the tip. I'm embarrassed I didn't think of trying that on my own!

if that does work... you SHOULD write a shell-script to cron up. (and link it here for those not wanting to make their own.)

for the rest of us that are tired of killing it manually randomly as we notice it.

it only takes up 100% of one (of eight) cores... but it is still incredibly annoying (and wrong) and ridiculous that it hasn't been fixed by 10.5.3.

does Apple really think server machines won't use SSH on a regular basis? seriously?

It seems like I just had the same problem with 10.5.3 on my MacBook Pro (Leopard non-server edition), but it did go away by itself. That is, I first killed the runaway sshd processes a few times, but they came back. Eventually I gave up and they stopped after a few minutes. There were some failed login attempts by bots at the time it happened.
My secure.log contains entries like this:
+com.apple.SecurityServer[21]: Failed to authorize right system.login.tty by client /usr/sbin/sshd for authorization created by /usr/sbin/sshd.+
And system.log contains:
+kernel[0]: devfs: ttys003: name slot allocation failed (Errno=17)+

It's exactly the same as described here, and sending the SIGSTOP helps, but this is extremely unpractical so I guess I'll reboot (and submit a bug report in the hopes it speeds up the fix).

I can confirm that after some time, attempts to launch an xterm, ssh session, or Terminal all result in lines like

Jun 19 17:07:25 Iparrizar kernel[0]: devfs: ttys004: name slot allocation failed (Errno=17)

filling my system.log. Clearly something is messed up with how Leopard is handing out virtual terminals that gets hung up and that hoses everything. The only solution I have found is to reboot, but that only seems to last about a week of my typical use before the problems manifest themselves again.

I noticed the issue for the first time a week or so ago.

Today: whilst connected via ssh to Mac OS X Server 10.5.3, and actively running a repozo* script at the server, I carelessly quit Terminal (or closed the Terminal window).

I was half-surprised that the Terminal window closed without a prompt while the remote process was running. (Is that normal?)

After making my mistake, I quickly connected via ARD to see whether Python etc. were still busy. AFAIR Activity Monitor at the server showed Python activity peaking for only a short while after my ARD connection.

It was then that I noticed the sshd CPU issue, coinciding with login attempts by

a) admin (the user that ran the repozo script)

b) root (the command was run with sudo).

(OT: a temporary file remained at the repozo destination. This temporary file was cleared when I re-ran the repozo script to completion.)

For as long as this bug exists: as a safeguard against my own carelessness, in
*Terminal preferences | Settings | Shell*
I have removed ssh from the list of processes that may be ignored when closing a window.

Thanks to Steffen M. for the
kill -SIGSTOP
tip, very useful.

I'll write to devbugs@apple.com re
rdar://5685756 and
rdar://5713758

Incidentally, Google results for the former also lead to
http://lists.apple.com/archives/Macos-x-server/2008/Mar/thrd6.html#00567

Graham

----
Repozo described at http://wiki.zope.org/ZODB/FileStorageBackup

Has anyone tried the 10.5.4 update to see if it corrects the problem? I don't see mention of the bug in the included updates. I'm getting this also, Xserve running 10.5.3

We're seeing a 100% CPU sshd process, and seeing "devfs: ttys000: name slot allocation failed (Errno=17)" in the log and we're on 10.5.4 server (build 9E17).

I did the 10.5.4 update last week, and it doesn't seem to have helped. If I don't reboot and let the system go, I've found that eventually (maybe unrelated) AppleFileServer will kernel panic the whole system, and then everything works good for awhile again.

Since updating Mac OS X Server from 10.5.3 to 10.5.4 the symptoms seem to be worse, more frequent.

Yesterday, multiple sshd processes causing maximum CPU activity. I restarted.

This morning, again, three sshd processes causing maximum CPU activity. I stopped each one of the three,

[omnium:~] root# sudo kill -SIGSTOP 9425
[omnium:~] root# sudo kill -SIGSTOP 2665
[omnium:~] root# sudo kill -SIGSTOP 2666

after which, I noted three other processes running riot:

FileSyncAgent
FileSyncAgent
FileSyncAgent

after those three ran their course,
AppleVNCServer
remained busy for a while (I had also connected via ARD) but then things seemed to settle.

Message was edited by: Graham Perrin

… and in my system.log files, various occurrences of

… name slot allocation failed (Errno=17)

Also, if it's relevant, more than a few

… Host at by will be blocked for at least 15.00 minutes

— without an IP address; where comparable lines appear with an IP address, most recently, they seemed to coincide with my office neighbour using Mail.app but the odd thing is/was, they don't have their Mail.app configured to use this server.

See selection pasted to http://pastie.textmate.org/231397