threading.Thread against multiprocessing.Process

Question

So recently I had the following problem : I have to make a server that handle request in order to update some values while the main process is using this values. So here, the server handling function is in the subprocess, and I can't stop it when i want.

In order to test what was the best solution for my problem between threading.Thread or multiprocessing.Process, I made this little following program :

import multiprocessing
import time
import threading

class SubProcess(multiprocessing.Process):

    def __init__(self, source):
        self.source = source
        super(SubProcess, self).__init__()

    def run(self):
        while 1:
            time.sleep(1) # Waiting for request...
            self.source.somevar["anotherkey"].append(5)
            print "My subprocess : ", id(self.source), id(self.source.somevar), self.source.somevar

class SubThread(threading.Thread):

    def __init__(self, source):
        self.source = source
        super(SubThread, self).__init__()

    def run(self):
        while 1:
            time.sleep(1) # Waiting for request...
            self.source.somevar["anotherkey"].append(5)
            print "My subthread : ", id(self.source), id(self.source.somevar), self.source.somevar

class Source:

    def __init__(self):
        self.somevar = {"akey": "THE key", "anotherkey": [5]}

    def start_process(self):
        self.process = SubProcess(self)
        self.process.start()

    def stop_process(self):
        self.process.terminate()

    def start_thread(self):
        self.thread = SubThread(self)
        self.thread.start()

    def stop_thread(self):
        # self.thread.stop() # What the hell should i put here
        pass

s = Source()

s.start_process()
time.sleep(2)
print "End : ", id(s), id(s.somevar), s.somevar
s.stop_process()

s.start_thread()
time.sleep(2)
print "End : ", id(s), id(s.somevar), s.somevar
s.stop_thread() # Obviously, thread never ends...

So threading.Thread modify the original s.somevar but I can't stop it, while multiprocessing.Process doesn't modify the original s.somevar but i can stop it.

I'm looking for a solution where I can stop the thread (with a SIGTERM) and where the thread can modify the original class Source, using the standard library. Is there any solutions ?

Answer 1

No, there isn't. See Is there any way to kill a Thread in Python? for a discussion of options - but these are co-operative! If that's not possible, use multiprocessing + messages to update a copy of the state locally.

Answer 2

To kill the thread, you need do something co-operative between the sub-thread and the main thread. With your example code, you could use a threading.Event:

class SubThread(threading.Thread):

    def __init__(self, source):
        self.source = source
        self.should_stop = threading.Event()
        super(SubThread, self).__init__()

    def run(self):
        while not self.should_stop.wait(1):
            #time.sleep(1) # No need to sleep, since we're waiting for 1 second above.
            self.source.somevar["anotherkey"].append(5)
            print "My subthread : ", id(self.source), id(self.source.somevar), self.source.somevar

    def stop(self):
        """ Call this to abort the thread. """
        self.should_stop.set()

This isn't as immediate a termination as it can be with process.terminate(), since you have to actually hit the call to should_stop.wait() before the thread is stopped.

To make the SubProcess work correctly, you need to use a process-safe shared variable. The multiprocessing module provides multiprocessing.Manager for this; it allows you to create shared variables in a manager process. The way you're updating the dict is actually a little tricky to handle properly though, because of some limitations with changing mutable values inside of the Proxy objects you get back from the Manager (see the note from the on that here). You have to explicitly re-assign the updated list to the dict for the dict to be updated properly:

class SubProcess(multiprocessing.Process):

    def __init__(self, source):
        self.source = source
        super(SubProcess, self).__init__()

    def run(self):
        while 1:
            time.sleep(1) # Waiting for request...
            # Can't do this with a Manager.dict
            #self.source.somevar["anotherkey"].append(5)

            # Do this instead. You'd need to do it with SubThread.run, too.
            l = self.source.somevar["anotherkey"]
            l.append(5)
            self.source.somevar["anotherkey"] = l
            print "My subprocess : ", id(self.source), id(self.source.somevar), self.source.somevar

class Source:

    def __init__(self):
        self.m = multiprocessing.Manager()
        # somevar is now process-safe.
        self.somevar = self.m.dict({"akey": "THE key", "anotherkey": [5]})

    # The rest is the same