Run Python script forever, logging errors and restarting when crashes

Question

I have a python script that continuously process new data and writes to a mongodb. In the script, its a while loop and a sleep that runs the code continuously.

What is the recommended way to run the Python script forever, logging errors when they occur, and restarting when it crashes?

Will node.js's forever be suitable? I'm also running node/meteor on the same Ubuntu server.

Answer 1

Recently wrote something similar. The basic pattern I follow is

while True:
    try:
         #functionality
    except SpecificError:
        #log exception
    except: #catch everything else
    finally:
        time.sleep(600)

to handle reboots you can use init.d or cron jobs.

Answer 2

If you are writing a daemon, you should probably do it with this command: http://manpages.ubuntu.com/manpages/lucid/man8/start-stop-daemon.8.html

You can spawn this from a System V /etc/init.d/ script, or use Upstart which is slowly replacing it.

Upstart: http://upstart.ubuntu.com/getting-started.html

System V: http://www.cyberciti.biz/tips/linux-write-sys-v-init-script-to-start-stop-service.html

I find System V easier to write, but if this will ever be packaged and distributed in a debian file, I recommend writing an Upstart conf.

Definitely keep the sleep so it won't keep a grip on CPU load.

Answer 3

supervisord is perfect for this sort of thing. While I used to check that programs were still running every couple of minutes with a cron job, supervisord runs all programs in an in-process thread, so in the event your program terminates, supervisord will automatically restart the process. I no longer need to parse the output of ps to see if a program crashed.

It has a simple declaritive config file and configurable logging. By default it creates a log file for your-program-name-stderr.log your-program-name-stdout.log which are automatically handled by logrotate when supervisord is installed from an OS package manager (Debian for me).

If you don't want to configure supervisord's logging, you should look at logging in python so you can control what goes into those files.

if you're on a debian derivative you should be able to install and start the daemon simply by executing apt-get install supervisord as root.

The config file is very straightforward too:

[program:myprogram]
command=/path/to/my/program/script 
directory=/path/to/my/program/base
user=myuser
autostart=true
autorestart=true
redirect_stderr=True

supervisorctl also allows you to see what your program is doing interactively and can start and stop multiple programs with supervisorctl start myprogram etc

Answer 4

I don't know if this is still relevant to you, but I have been reading forever about how to do this and want to share somewhere what I did.

For me, the goal was to have a python script running always (on my Linux computer). The python script also has a "while True " loop in it which should theoretically run forever, but if it for any reason I cannot think of would crash, I want the script to restart. Also, when I restart the computer it should run the script.

I am not an expert but for me the best and most understandable was to use systemd (assuming you use Linux).

There are two nice examples of how to do this given here and here, showing how to write your .service files in either /etc/systemd/system or /lib/systemd/system. If you want to be completely correct you should take the former:

" /etc/systemd/system/: units installed by the system administrator" 1

The documentation of systemd here is actually nice to read, even if you are not an expert.

Hope this helps someone!