10.6.2 httpd respawning endlessly.

Question

10.6.2 httpd respawning endlessly.

My server's log is full of this:
Feb 28 00:40:31 atlas com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds
Feb 28 00:40:41 atlas org.apache.httpd[2466]: (48)Address already in use: make_sock: could not bind to address 127.0.0.1:9010
Feb 28 00:40:41 atlas org.apache.httpd[2466]: no listening sockets available, shutting down
Feb 28 00:40:41 atlas org.apache.httpd[2466]: Unable to open logs
Feb 28 00:40:41 atlas com.apple.launchd[1] (org.apache.httpd[2466]): Exited with exit code: 1

Which would of course suggest that something else has that port. It doesn't. The only thing with that port is httpd. So I kill all running instances of httpd, nothing has the port. Start up Web ... and the errors begin churning out again.

When I reboot the server, there's nothing on port 9010 (confirmed by lsof). Start Web, and the errors begin churning out again.

So I edit httpd.conf, and change the listen line to 9011. Start web, now it's complaining it can't bind to :9011! It can't bind, because it itself has just bound!

I comment out the Listen line entirely, and now the error is:
Feb 28 00:43:14 atlas org.apache.httpd[2621]: (48)Address already in use: make_sock: could not bind to address [::]:80
Feb 28 00:43:14 atlas org.apache.httpd[2621]: (48)Address already in use: make_sock: could not bind to address 0.0.0.0:80

This is an absolutely stock install. I haven't touched Web -- all I did was upgrade to 10.6.2. To be sure, I deleted httpd.conf and the sites and restored from their defaults but still this error is haunting me.

It's incredibly frustrating, and I'd very much appreciate any help you can give.

Mac mini server, Mac OS X (10.6.2)

Posted on Feb 27, 2010 4:48 PM

Reply

Answer 1

MrHoffman

Level 10

146,251 points

Feb 27, 2010 6:13 PM in response to Allan Donald1

Verify that there isn't anything already yakking on port 80, too.

+$ sudo lsof -i -P | grep -i ":80"+

Mayhap a firewall setting?

Multiple virtual hosts tapped onto port 80?

Reply

Answer 2

Feb 28, 2010 5:54 AM in response to MrHoffman

Thanks for the reply. :80 came up as I expected: nothing after a reboot, then httpd after I start web.

I don't think it can be virtual hosts, unless it comes configured like that -- this really is effectively a stock install.

Reply

Answer 3

MrHoffman

Level 10

146,251 points

Feb 28, 2010 7:57 AM in response to Allan Donald1

Digging around in the log files, try an apache configtest, and, well, figure out where this configuration diverges from stock, are reasonable steps forward here.

Please don't mentally decide that this is a "stock install" or that this is not virtual hosts. While that might well actually be the case here, debugging these involves ruling triggers in and ruling triggers out. Assumptions can mask triggers.

Reply

Answer 4

Feb 28, 2010 8:03 AM in response to MrHoffman

It's not so much as a mental decision as it is the fact that the machine was formatted and installed yesterday morning! The only changes between then and the problem were the upgrade to 10.6.2, the creation of two users in the OD, and creating three MySQL databases (with the system MySQL install).

Apache configtest passes, a diff between httpd.conf and httpd.conf.default says they're identical. The sites in the sites folder are also identical to their .defaults.

The log files don't have anything beyond what I put in my first post; the error_log and access_logs themselves don't have anything unusual.

Thanks for the suggestions, however.

Reply

Answer 5

MrHoffman

Level 10

146,251 points

Feb 28, 2010 9:01 AM in response to Allan Donald1

Something here is not stock. (Or Mac OS X Server 10.6.2 is badly broken.) Verified the DNS with +changeip -checkhostname+ on the box, and with dig?

Reply

Answer 6

Mar 4, 2010 2:17 PM in response to MrHoffman

Yep, DNS checks out fine.

Basically, it feels as though launchd is trying to respawn new instances of httpd, when they're already running. If I:

1. Restart the box (with Web set to stopped in server admin), then there's nothing running on port 80 or port 9010 after startup.
2. Start the Web service, it'll start up and serve sites fine.
3. Then the console log will fill up with new instances of apache trying to bind to ports it can't have, because a legit copy of apache already owns them.
4. If I then "sudo launchctl unload org.apache.plist", the log stops filling up, but apache keeps on running fine (even spawing new server processes as required).

So: what is going wrong that means launchd doesn't know httpd started up successfully?

Thanks for any help!

Reply

Answer 7

Mar 4, 2010 2:25 PM in response to Allan Donald1

Ah, here we go, that's exactly what it is.

On this system, /System/Library/LaunchDaemons/org.apache.httpd.plist contains

[lines snipped]
<key>ProgramArguments</key>
<array>
<string>/usr/sbin/httpd</string>
<string>-D</string>
<string></string>
</array>

What it should have (or at least, what 10.6 non-server and what works) is to have:
<key>ProgramArguments</key>
<array>
<string>/usr/sbin/httpd</string>
<string>-D</string>
<string>FOREGROUND</string>
</array>

This stops Apache daemonizing, which is neccessary for launchd to be able to keep track of it.

(It'd be of interest if people could report what their plists look like, to see if this has somehow been corrupted, or if it's a 10.6.2 bug that needs reporting)

Reply

Answer 8

MrHoffman

Level 10

146,251 points

Mar 4, 2010 3:56 PM in response to Allan Donald1

Looking at a test server that's close to your configuration, the plist contains FOREGROUND.

Reply

Answer 9

TigerKR

Level 1

59 points

Mar 30, 2010 6:43 PM in response to Allan Donald1

OMG! I've been trying to fix this for months! Thank you!!!

Adding FOREGROUND to the plist fixed the problem!

Thank you!!!

Reply