Waiting for WatchService event to be done processing before accessing the event file

Question

I am watching a directory using java.nio.file.WatchService; and every time a file is created I call processFile() passing it the File handle.

Here is the code for the watcher:

boolean poll = true;
System.out.println("Watching Directory: "+path.toString()+"...");

while (poll) {
    WatchKey key = watchService.take();

    for (WatchEvent<?> event : key.pollEvents()) {
        WatchEvent<Path> ev = cast(event);
        Path dir = (Path)key.watchable();
        Path fullPath = dir.resolve(ev.context());
        File file = new File(fullPath.toString());

        System.out.println("Event kind:" + event.kind()
                + ". File affected: " + event.context() 
                + ". Full path" + fullPath);

        if(event.kind() == ENTRY_CREATE) fileProcessor.processFile(file);
    }
    poll = key.reset();
}   

Here is the processFile() code:

public void processFile(File file) {
    String threadName = "Thread-" + file.getName();
    Thread t = ProcessorUtil.getThreadByName(threadName);

    if (t == null || !t.isAlive()) {
        new Thread(() -> {
            try {
                Thread.sleep(1000); //HACKY WAY of WAITING
                InputStream in = new FileInputStream(file);
                someProcess(in);
            } catch (Exception e) {
                _log.error("Exception: ",e);
            }
        }, threadName).start();
    } else {
        System.out.println("File "+file.getName()+"  currently being processes");
    }

As you can see, I'm waiting for the system to write the file before I can access the FileInputStream. This doesn't seem right to me, and I wonder if there is a better way of doing this than calling Thread.sleep().

Answer 1

The watchservice can send you many events together and might even give CREATE then MODIFY events for same file, or DELETE, CREATE, MODIFY all in same list of events for files being written elsewhere (eg text editor re-writes).

The easiest way to protect from mid-process writes and reports to wait until the watch service pauses. Your logic needs to ws.take() for first query, and then ws.poll(1) if you've already received events:

WatchKey wk = hasPending ? ws.poll(1,TimeUnit.MILLISECONDS) : ws.take();

This is snippet from a test program which collates all events and acts on them when no watch results are returned by the last poll(1).

Set<Path> created = new LinkedHashSet<>();
Set<Path> modified = new LinkedHashSet<>();
Set<Path> deleted = new LinkedHashSet<>();

while(appIsRunning) {
    boolean hasPending = created.size() + modified.size() + deleted.size() > 0;
    WatchKey wk = hasPending ? ws.poll(100,TimeUnit.MILLISECONDS) : ws.take();

    if (wk != null)  {
        for (WatchEvent<?> event : wk.pollEvents()) {
             Path parent = (Path) wk.watchable();
             Path eventPath = (Path) event.context();
             store(event.kind(), parent.resolve(eventPath), created, modified, deleted);
         }
         boolean valid = wk.reset();
         if (!valid) {
             throw new RuntimeException("Check the path, it may be deleted: "+dir);
         }
    }

    System.out.println("PENDING: cre="+created.size()+" mod="+modified.size()+" del="+deleted.size());

    // ONLY HANDLE EVENTS IF NOTHING WAS RECEIVED:
    if (wk == null && hasPending) {
        // FIRE EVENTS for each list HERE: deleted, created, modified

        // reset the list for next take() cycle:
        deleted.clear();  created.clear(); modified.clear();
    }
}

If you're only catching CREATE events the store() is easy, this version collates multiple kinds to latest sensible type:

/**
 * Save event for later processing by event kind EXCEPT for:
 * <li>DELETE followed by CREATE           => store as MODIFY
 * <li>CREATE followed by MODIFY           => store as CREATE
 * <li>CREATE or MODIFY followed by DELETE => store as DELETE
 */
private static void
store(Kind<?> kind, Path path, Set<Path> created, Set<Path> modified, Set<Path> deleted) {
    System.out.println("STORE "+kind+" path:"+path);

    boolean cre = false;
    boolean mod = false;
    boolean del = kind == StandardWatchEventKinds.ENTRY_DELETE;

    if (kind == StandardWatchEventKinds.ENTRY_CREATE) {
        mod = deleted.contains(path);
        cre = !mod;
    }
    else if (kind == StandardWatchEventKinds.ENTRY_MODIFY) {
        cre = created.contains(path);
        mod = !cre;
    }
    addOrRemove(created,  cre, path);
    addOrRemove(modified, mod, path);
    addOrRemove(deleted,  del, path);
}

// Add or remove from the set:
private static void addOrRemove(Set<Path> set, boolean add, Path path) {
    if (add) set.add(path);
    else     set.remove(path);
}

This cuts down the chance that your event handler for CREATE will clash with the writing operation or miss the end of the file.