Python watchdog: Wait until file output from different programs to watched directory is complete before further processing

Question

I have several dockerized applications that output data to a common directory and a dockerized python watchdog script that watches that directory and is supposed to process the data further once the program output is complete.

It's essential that a file must be complete before it's processed further, I tried to implement a solution similar to Python watchdog windows wait till copy finishes :

def on_modified(event):
    sizeBefore = -1
    while os.path.getsize(event.src_path) != sizeBefore:
        sizeBefore = os.path.getsize(event.src_path)
        time.sleep(1)
    print(f"[+] File transfer of {event.src_path} completed")

which works ok for single files but breaks when it has to monitor multiple files. Waiting until program output is complete and then copy a file to the watched folder is also not an option as I'm dealing with potentially large files that will take at least a few seconds to copy and then I'm having the same issue.

Any ideas on how I can adjust my on_modified event handler to properly watch until a file is not modified anymore, working for multiple files in a single directory