Python Watchdog: Is there a way to pause the observer?

Question

I am using Watchdog to monitor a directory and keep it in sync with Dropbox.

I am facing a situation where every time I download a file from Dropbox, I trigger an upload event as I need to write to the directory Watchdog is monitoring. This is the code I am using.

event_handler = UploadHandler.UploadHandler()
observer = Observer()
observer.schedule(event_handler, path=APP_PATH, recursive=True)
observer.start()

try:
    while True:
        # Apply download here   
        time.sleep(20)

except KeyboardInterrupt:
    observer.stop()

observer.join()

Is there a way to "pause" the observer while I apply the download and "unpause" it again when I'm done?

Answer 1

You can overwrite the method dispatch_events of the class Observer to skip events you do not want to dispatch. Here is an example how to do it:

class SkipObserver(watchdog.observers.Observer):
    def __init__(self, *args):
        Observer.__init__(self, *args)
        self._skip_list = []

    def skip(self, event):
        with self._lock:
            self._skip_list.append(event)

    def dispatch_events(self, event_queue, timeout):
        event, watch = event_queue.get(block=True, timeout=timeout)
        try:
            if event in self._skip_list:
                self._skip_list.remove(event)
            else:
                self._dispatch_event(event, watch)
        except KeyError:
            pass
        event_queue.task_done()

Now replace Observer by SkipObserver in your example and use the method skip to skip events. Beware of using skip in a handler since this would end with a deadlock, skip is blocking as implemented here. While handling an event the observer is locked.

If you want to specify skipping for every handler, you can use an analogous approach to the above one.

Answer 2

I needed pausing functionality so I'm using the following observer:

import time
import contextlib
import watchdog.observers


class PausingObserver(watchdog.observers.Observer):
    def dispatch_events(self, *args, **kwargs):
        if not getattr(self, '_is_paused', False):
            super(PausingObserver, self).dispatch_events(*args, **kwargs)

    def pause(self):
        self._is_paused = True

    def resume(self):
        time.sleep(self.timeout)  # allow interim events to be queued
        self.event_queue.queue.clear()
        self._is_paused = False

    @contextlib.contextmanager
    def ignore_events(self):
        self.pause()
        yield
        self.resume()

I can then pause the observer directly with the pause() and resume() methods, but my primary use case is for when I just want to ignore any events caused by writing to a directory I'm watching, for which I use the context manager:

import os
import datetime
import watchdog.events


class MyHandler(watchdog.events.FileSystemEventHandler):
    def on_modified(self, event):
        with OBSERVER.ignore_events():
            with open('./watchdir/modifications.log', 'a') as f:
                f.write(datetime.datetime.now().strftime("%H:%M:%S") + '\n')

if __name__ == '__main__':
    watchdir = 'watchdir'
    if not os.path.exists(watchdir):
        os.makedirs(watchdir)

    OBSERVER = PausingObserver()
    OBSERVER.schedule(MyHandler(), watchdir, recursive=True)
    OBSERVER.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        OBSERVER.stop()
    OBSERVER.join()

You can test this out by saving both code blocks in a file, running it, and adding/editing/removing files in the created 'watchdir' directory. The timestamps of your modifications will be appended to 'watchdir/modifications.log'.

This functionality appears to be built into pyinotify, but that library only works in linux and watchdog is OS-independent.

Answer 3

After spending a lot of time (5am now, apologies for any typos and stuff) with both answers here, to no avail, I felt stupid and realized a much simpler solution for my needs.

Summary (code below)

1. I'm not using a custom Observer, but you can.
2. Make sure your Event Handler is in the same file that you're declaring your Observer in.
3. Create a global called PAUSED or something similar
4. Inside on_modified or whatever event function you're using inside your Event Handler, check if PAUSED is True.
5. If True, exit, if not, continue and pause. Then resume when done.

Put this at the top of the file:

global PAUSED
PAUSED = False

Put this in EventHandler.on_modified() or whatever your event handler is, and whatever event function you're using:

# Retrieve global
global PAUSED

# If PAUSED, exit
if PAUSED:
  return

# If not, pause anything else from running and continue
PAUSED = True

# Do stuff here

# Once finished, allow other things to run
PAUSED = False

Reasoning: This will completely ignore anything that tries to run during the pause. I needed this because Watchdog will fire up to 5 times per "file modified" event because the file is still being written. This ensures it only fires once, pauses, and doesn't resume until those other duplicate events have expired.

Answer 4

To pause the observer while handling events (for example in the case the event handler modifies the file):

class PausableObserver(Observer):
    @contextmanager
    def pause(self):
        orig_put = self.event_queue.put
        self.event_queue.put = lambda x : None
        try:
            yield
        finally:
            self.event_queue.put = orig_put

usage:

with observer.pause():
    modify_file()
    sleep(.1)

The other methods here would keep the events on queue as there is no concurrent dispatch of events.