Problems with multiprocess-watchdog script in Python 3

260 Views Asked by At

I have a watchdog-script to monitor a directory recursively. When an event happens I want to do something with the file.

This worked super fine, but some files are very big, so the treatment of this file blocked the watcher and some files were later missing because the watcher didn't recognize them due to the blocking.

So I thought, multiprocessing could help. My idea was, that the event (created, modified, ....) would start a new process and then execute the function.

I do now have a sample script that combines watchdog with multiprocessing, but I am having trouble getting it working.

import os
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler
from multiprocessing import Process


def print_msg(text):
    proc = os.getpid()
    print("{0} über Prozess ID: {1}".format(text, proc))

def on_created(event):
    text = "hey, {0} has been created!".format(event.src_path)
    proc = Process(target=print_msg, args=(text))
    proc.start()
    proc.join()

def on_deleted(event):
    text = "what the f**k! Someone deleted {0}!".format(event.src_path)
    proc = Process(target=print_msg, args=(text))
    proc.start()
    proc.join()

def on_modified(event):
    text = "hey buddy, {0} has been modified".format(event.src_path)
    proc = Process(target=print_msg, args=(text))
    proc.start()
    proc.join()

def on_moved(event):
    text = "ok ok ok, someone moved {0} to {1}".format(event.src_path, event.dest_path)
    proc = Process(target=print_msg, args=(text))
    proc.start()
    proc.join()

if __name__ == "__main__":
    patterns = "*"
    ignore_patterns = ""
    ignore_directories = False
    case_sensitive = True
    my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)
    
my_event_handler.on_created = on_created
my_event_handler.on_deleted = on_deleted
my_event_handler.on_modified = on_modified
my_event_handler.on_moved = on_moved

path = "\\\swibinacl01-cifs\\BelVis\\PROD\\Importer\\Messdaten"
go_recursively = True
my_observer = Observer()
my_observer.schedule(my_event_handler, path, recursive=go_recursively)

my_observer.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    my_observer.stop()
    my_observer.join()

When I test the script I get following message just when the first event comes up:

my_event_handler.on_created = on_created
NameError: name 'my_event_handler' is not defined

So I think that after the event (or starting the new process) the my_event_handler object has gone and needs to be re-initialized.

But why is that? My thinking was, that when the event starts the function within a new process, the original process (the watcher) would continue and the new process acts independently.
What is my mistake? Can anyone help me out?

0

There are 0 best solutions below