6 Replies Latest reply: May 18, 2013 6:30 AM by mario.d
mario.d Level 1 Level 1 (0 points)

This is really strange and I almost don't know where to begin.

 

I'm running a small server written in Python (using the Bottle framework) and launching it using launchd. The server may run fine all day long, or it may unexpectedly stop serving requests during the day. However, the process still appears to be running. I can see the process these two different ways:

 

  • ps aux
  • launchctl list

 

If I simply stop the process in launchctl and then start it again, everything is fine. But until I restart it, nothing happens. When everything is running correctly, the server logs to a file. Unfortunately, there is no indication in the server's log file that anything has gone wrong—you simply see the last request and nothing after that. I saw nothing in the System logs to suggest there is anything wrong.

 

When I launch the server by itself, manually (forgoing the use of launchd/launchctl), this doesn't seem to happen. (I have not proved this out through extensive testing though.) The reason I'm using launchctl is because I would like to run the server as an unprivileged user (_www). Could that have anything to do with it?

 

Anyone know of a way I can troubleshoot this? Is there another log I should be looking at? Anyone run across something like this before? Anyone have any questions?

 

Just to reiterate, an entire day (maybe more) has gone by without any issue, but usually when I run the thing, it will stop serving requests at some random point during the day.

 

Thanks!

  • 1. Re: Server process controlled by launchd stops serving
    MrHoffman Level 6 Level 6 (12,470 points)

    Add more logging to your Python code?

  • 2. Re: Server process controlled by launchd stops serving
    Frank Caggiano Level 7 Level 7 (23,875 points)

    In addition to MrHoffman's suggestion how about showing us your plist file.

  • 3. Re: Server process controlled by launchd stops serving
    mario.d Level 1 Level 1 (0 points)

    I may very well be failing to understand something about launchctl plist files. Thanks!

     

    <?xml version="1.0" encoding="UTF-8"?>

    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

    <plist version="1.0">

    <dict>

        <key>RunAtLoad</key>

        <true/>

        <key>KeepAlive</key>

        <dict>

            <key>SuccessfulExit</key>

            <true/>

        </dict>

        <key>Label</key>

        <string>com.mariodiana.receiptsserver</string>

        <key>UserName</key>

        <string>_www</string>

        <key>ProgramArguments</key>

        <array>

            <string>/usr/bin/python</string>

            <string>/opt/www/Receipt_proj/receipts_webapp.py</string>

            <string>192.168.0.100</string>

        </array>

        <key>WorkingDirectory</key>

        <string>/opt/www/Receipt_proj</string>

        <key>StandardErrorPath</key>

        <string>/opt/www/Receipt_proj/log/out.log</string>

        <key>Sockets</key>

        <dict>

            <key>Listeners</key>

            <dict>

                <key>SockServiceName</key>

                <string>http-alt</string>

                <key>SockType</key>

                <string>stream</string>

                <key>SockFamily</key>

                <string>IPv4</string>

            </dict>

        </dict>

    </dict>

    </plist>

  • 4. Re: Server process controlled by launchd stops serving
    mario.d Level 1 Level 1 (0 points)

    I found this posting on Stack Overflow that leads me to believe the problem is primarily with bottle.py, the Python framework I'm using. However, there does seem to be one thing with launchd that is triggering this problem, and I'd like some feedback. First, here's the Stack Overflow post:

     

    http://stackoverflow.com/questions/14444456/bottle-py-wsgi-server-stops-respondi ng

     

    And here is the part from there that has got me thinking:

     

    The only way I have been able to reproduce the issue is when I try to run a second script on my Ubuntu server that creates another server listening to a different port (ie.: exactly the same script as above but port=3001). If I send a request to the newly created server, I also do not get a response and the connection eventually times out.

     

    Here's the thing—launchd seems to be firing off two processes. I've confirmed this by running "ps aux" in Terminal. That seems similar to what the guy in the post quoted is doing manually. Does anyone know why there would be two processes launched by launchd? Is there a way to restrict it to one process?

     

    Thanks!

  • 5. Re: Server process controlled by launchd stops serving
    Linc Davis Level 10 Level 10 (118,505 points)

    Your job will only be restarted if it exits with status zero. If it crashes, it won't be restarted. Also, you specify launch-on-demand from the http-alt port, but you also specify run at load. They're inconsistent.

     

    If your script does not daemonize or fork, you should remove the RunAtLoad and KeepAlive keys. If it's unable to accept the socket handoff, which is likely, you should also remove the Sockets key.

     

    Technical Note TN2083

    Creating Launch Daemons and Agents

    launchd.plist(5) Mac OS X Manual Page

  • 6. Re: Server process controlled by launchd stops serving
    mario.d Level 1 Level 1 (0 points)

    THIS SOLVED MY QUESTION

     

    Sorry, I clicked "This helped me" because your post was very informative, and I figured that I should give it a few days to see if the server's behavior changed after making your suggested changes. I have not had the problem since making the changes you suggested, and returned to indicate that your post solved my question. However, it seems that once you click "This helped me" the "This solved my question" choice is no longer available.

     

    I would mark your reply properly if I could. Thanks again!