Safely interrupt a program with threaded downloading

Question

I have a (commandline/terminal) program that scrapes a website with worker threads that do the downloading from a queue and a main thread that downloads the index files (50 entries per page). How can I make the program check for an interrupt (either CTRL+C or my own defined one), and when it catches such interrupt it will first clean up (download the remaining queue) and then terminate.

Answer 1

Wrap the main function that's waiting for the threads to complete with an exception handler like this:

try:
    main()
except KeyboardInterrupt:
    stop()

def stop():
    for t in threads:
        t.my_stop_function()
    # wait for threads to stop here...

class MyThread(threading.Thread):
    def __init__(self, *args, **kwargs):
        threading.Thread.__init__(self, *args, **kwargs)
        self.stop = False

    def my_stop_function(self):
        self.stop = True

    def run(self):
        while not self.stop:
            scrape()

Answer 2

In the main loop, you want to catch KeyboardInterrupt exceptions (raised when user press CTRL-C). For handling cleanup, you can use atexit module to run some a global cleanup function or use threading.Event/threading.Condition to notify the worker threads to cleanup themselves and exit.

import atexit
atexit.register(cleanup_function)