SSHD Processes Using 100% CPU

If anything, it's far worse with 10.5.4. 10.5.3 seemed to improve the situation slightly, but 10.5.4 seems to be a big step backwards. Often it takes many attempts to get an SSH connection, leaving lots of sshd processes using 100% CPU.

Once logged in, we can run a little script to kill off the bad sshds spawned in all the failed login attempts..

#!/opt/local/bin/php
<?php

$output = `ps -aTrxx | egrep '([0-9]+) +\?\? +[0-9]+\:[2-9][0-9]\.[0-9]{2} /usr/sbin/sshd -i'` ;
$numMatches = preg_match_all('/^\s*([0-9]+)\s+/m',$output,$matches,PREG_SET_ORDER) ;

if ($numMatches > 0) {
foreach($matches as $key => $match) {
$killCmd = 'kill -9 '.$match[1] ;
$handle = popen($killCmd, 'r');
pclose($handle);
}
}
print_r($matches) ;


... but we really shouldn't have to be doing this. I'm seriously toying with downgrading this box to 10.4.x as the machines we have on that have been absolutely rock solid compared with 10.5.x.

I have just ran into this problem on a 10.5.2 Server.

Perhaps we need to brute-force Apple into fixing this? What if everyone on this thread emailed devbugs at the same time asking about the same radar issues? It might give pause to whoever checks the email for devbugs when they realize they have been flooded with inquiries about the same issue.

How about it? Should we coordinate a mass-email to devbugs?

I don't know the etiquette but rather than co-ordinate a mass e-mail, it could be more appropriate for

a) each one of us to file a bug report with proper information

and

b) Apple to collate information from the reports and decide upon escalation.

On one hand: I'm focused on the comments from Chris Adams, in particular

openpty() syscall and that's where the actual failure occurs

— without understanding the technical aspects, I assume that this may be common to the *range of circumstances* in which we see 100% CPU activity.

On the other hand: I'm aware that in my own cases, the symptoms were first noted after user error (my own)

sshd issue after inappropriate closure of a Terminal win

but with less obvious cause on other occasions, most recently

FileSyncAgent processes using 100% after sudo kill -SIGSTOP sent to sshd

— so maybe some other factors are contributory to what we perceive to be the issue.

I'm treating this as a reminder to myself … good practice bug reporting…

Regards
Graham

When this happens to me I can force quit SSHD but it never properly respawns. I also can't launch Terminal or X11 without it also pegging the processors. Clearly this is not an attack on SSH but something to do with creating terminals, local or virtual. Once this happens, I don't know of any way to get command line access other than a reboot because I can't launch Terminal. Any ideas on workarounds?

Well, it seems that it is related to attacks on SSH. I have found that unless an attack is launched against SSH, the problems you are describing (can't launch Terminal, X11, or anything else for that matter) don't occur.

I took a look through the logs on our server and created firewall rules that block the logged IPs from communicating with the SSH ports. I also enabled the logging of these denied packets. So far, the server still works. I can launch apps and SSH into it myself. There are no entries of denied packets in the firewall log. So either firewall logging doesn't work correctly, or I have been fortunate in that no other attacks were launched against the server.

This workaround just worked for me, on OS X 10.5.4. Thanks for the tip!

I'm having this exact same issue and even saw httpd hang at 99% CUP yesterday! I tried what "Zeppomedio" wrote and I get the same result! I feel like I'm on a windows machine!

I just installed "Security Update 2008-005" on 8/5/2008 and am still seeing sshd hang at 100% and terminal at 100%!

I'm running 10.5.4 and Security Update 2008-005 and still having the same issues. 😟

I have been able to log into SSH when I load terminal, while it's hanging, ssh remotely and it works!! Of course, then you have to force quit terminal. Very bizarre!

Jonathan Abrams wrote:
Well, it seems that it is related to attacks on SSH. I have found that unless an attack is launched against SSH, the problems you are describing (can't launch Terminal, X11, or anything else for that matter) don't occur.

I took a look through the logs on our server and created firewall rules that block the logged IPs from communicating with the SSH ports. I also enabled the logging of these denied packets. So far, the server still works. I can launch apps and SSH into it myself. There are no entries of denied packets in the firewall log. So either firewall logging doesn't work correctly, or I have been fortunate in that no other attacks were launched against the server.

Thank you for this tip. I implemented a firewall rule that restricts all access to my server from any address other than my home machines, and after rebooting, have yet to witness the sshd hang. What a weird bug. It certainly smells like a successful DoS attack against OS X sshd, doesn't it?

steve maller wrote:
Thank you for this tip. I implemented a firewall rule that restricts all access to my server from any address other than my home machines, and after rebooting, have yet to witness the sshd hang. What a weird bug. It certainly smells like a successful DoS attack against OS X sshd, doesn't it?

You are welcome. It does, but Apple seems too focused on its iGizmos and MobileMe to do anything about it. I suggested earlier in this thread that we all flood the devbugs@apple.com email address at the same time on the same day asking about the status of this bug. My thinking was that if we all flood that address at the same time about the same bug, perhaps whoever is checking that email account will say "Hmm...maybe we should do something about this sooner rather than later?"

Unfortunately at least one person was against this idea, and I never got any traction from anyone else.

Message was edited by: Jonathan Abrams for grammar

Jonathan Abrams wrote:

it seems that it is related to attacks on SSH

Not always. In my own cases, the symptoms were first noted after user error (my own):

sshd issue after inappropriate closure of a Terminal window

On 14th July I wrote:

it could be more appropriate for a) each one of us to file a bug report with proper information

My own bug ID 6065930 was determined by Apple to be a duplicate of 5685756.

I'm all for it.. I emailed them about it last week.. This was my reply:

Thank you for contacting us regarding the status of Bug ID# 5685756.

At this time, there isn't any new information available for this issue. I have checked with engineering and the issue is still being investigated.

Please be sure to regularly check the seed or release notes for potential or related fixes that might affect this issue.

We sincerely appreciate your patience and thank you for your support.


*So, name the date and time and I'll do it with you.. Anyone else?*

Message was edited by: jgalicinski

Today, and on two or three occasions previously, clients are presented with the following error when they attempt an AFP connection to our Mac OS X Server 10.5.4:

Connection failed

This file server will not allow any additional users to
log on. Please try to connect again later.

Whilst I can't estimate the time today when that behaviour began, it was certainly apparent (I checked for the symptom) after I noticed and attended to a CPU-hungry sshd process.

Re http://discussions.apple.com/message.jspa?messageID=7358097#7358097 and Steffen M.'s

kill -SIGSTOP <process-ID of_the_hangingSSHD>

I sent the SIGSTOP signal to the sshd process.

Then focused on just two of the few AFP client computers.

Restarting client 'A' did not resolve the AFP 'Connection failed' issue.

Restarting client 'B' did seem to resolve the issue; from a third computer I could make AFP connections. If it's relevant, client 'B' has its home directory on the AFP server.

However: later the AFP 'Connection failed' symptom returned, and even after client 'B' was shut down I could not make new AFP connections.

At some point I wondered whether sending the SIGCONT signal to the stopped sshd process would help but apparently not, still it was not possible to make new AFP connections.

Ultimately, around 17:00 UK time I opted to restart the Mac OS X Server 10.5.4 and in the few hours since then it has been OK. Tomorrow and beyond, I'll keep a close eye on not only sshd processes but also AFP connectivity.

*Relevance to this thread*

In my bug report to Apple I included:

… On one occasion I found that the server could not accept new AFP connections, presumably as a result of the activity/issue…

and that report overall was determined to be a duplicate of bug ID 5685756

jgalicinski wrote:
*So, name the date and time and I'll do it with you.. Anyone else?*

Tuesday at 11 PM EST? They should see it first thing Wednesday morning.