USN NFTS change notification event interrupt

Question

I'm trying to find a way to let the system tell me whenever there's a new entry in the USN Change Journal to track modifications made to files and directories on an NTFS volume (Server 2008/2012).

This way I don't have to constantly poll the journal and can just let my thread sleep until I get notified when there's a new change-event.

However, is there even such an interrupt?

The FSCTL_QUERY_USN_JOURNAL function doesn't specifically mention interrupts (events, notifications), nor have I been able to find another way to achieve this with less intensive poll-and-compare techniques.

I'm not a hard-core programmer so there may be simpler ways to tie these functions to interrupts that I'm not aware of.

Could I perhaps find out where the USN Change Journal is stored and watch that file with another process that can generate and interrupt on change?

https://msdn.microsoft.com/en-us/library/aa365729(v=vs.85).aspx

Answer 1

You can use Journal, but in this case I'd use easier method via registering a directory notification by calling the FindFirstChangeNotification or ReadDirectoryChangesW functions, see https://msdn.microsoft.com/en-us/library/aa364417.aspx

If you'd prefer to use Journal, this is - I think - the best introductory article with many examples. It is written for W2K, but those concepts are still valid: https://www.microsoft.com/msj/0999/journal/journal.aspx

Answer 2

The code posted here blocks the executing thread till the new USN record is created in the Journal. When new records arrive, the thread awakens and you can process changes and/or notify listeners via a callback that filesystem has changed (in the example it just prints message to the console). Then the thread blocks again. This example uses one thread per volume (so for each volume, separate NTFSChangesWatcher class instance needed).

It is not specified which tools or language you use, so I will write as I did it. To run this code, create a Visual Studio C++ Win32 Console Application. Create NTFSChangesWatcher class. Paste this code in NTFSChangesWatcher.h file (replacing auto-generated one):

#pragma once

#include <windows.h>
#include <memory>

class NTFSChangesWatcher
{
public:
    NTFSChangesWatcher(char drive_letter);
    ~NTFSChangesWatcher() = default;

    // Method which runs an infinite loop and waits for new update sequence number in a journal.
    // The thread is blocked till the new USN record created in the journal.
    void WatchChanges();

private:
    HANDLE OpenVolume(char drive_letter);

    bool CreateJournal(HANDLE volume);

    bool LoadJournal(HANDLE volume, USN_JOURNAL_DATA* journal_data);

    bool NTFSChangesWatcher::WaitForNextUsn(PREAD_USN_JOURNAL_DATA read_journal_data) const;

    std::unique_ptr<READ_USN_JOURNAL_DATA> GetWaitForNextUsnQuery(USN start_usn);

    bool NTFSChangesWatcher::ReadJournalRecords(PREAD_USN_JOURNAL_DATA journal_query, LPVOID buffer,
        DWORD& byte_count) const;

    std::unique_ptr<READ_USN_JOURNAL_DATA> NTFSChangesWatcher::GetReadJournalQuery(USN low_usn);


    char drive_letter_;

    HANDLE volume_;

    std::unique_ptr<USN_JOURNAL_DATA> journal_;

    DWORDLONG journal_id_;

    USN last_usn_;

    // Flags, which indicate which types of changes you want to listen.
    static const int FILE_CHANGE_BITMASK;

    static const int kBufferSize;
};

and this code in NTFSChangesWatcher.cpp file:

#include "NTFSChangesWatcher.h"

#include <iostream>

using namespace std;

const int NTFSChangesWatcher::kBufferSize = 1024 * 1024 / 2;

const int NTFSChangesWatcher::FILE_CHANGE_BITMASK =
   USN_REASON_RENAME_NEW_NAME | USN_REASON_SECURITY_CHANGE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_OVERWRITE |
   USN_REASON_DATA_TRUNCATION | USN_REASON_DATA_EXTEND | USN_REASON_CLOSE;


NTFSChangesWatcher::NTFSChangesWatcher(char drive_letter) :
    drive_letter_(drive_letter)
{
    volume_ = OpenVolume(drive_letter_);

    journal_ = make_unique<USN_JOURNAL_DATA>();

    bool res = LoadJournal(volume_, journal_.get());

    if (!res) {
        cout << "Failed to load journal" << endl;
        return;
    }

    journal_id_ = journal_->UsnJournalID;
    last_usn_ = journal_->NextUsn;
}

HANDLE NTFSChangesWatcher::OpenVolume(char drive_letter) {

wchar_t pattern[10] = L"\\\\?\\a:";

pattern[4] = static_cast<wchar_t>(drive_letter);
HANDLE volume = nullptr;

volume = CreateFile(
    pattern,  // lpFileName
    // also could be | FILE_READ_DATA | FILE_READ_ATTRIBUTES | SYNCHRONIZE
    GENERIC_READ | GENERIC_WRITE | SYNCHRONIZE,              // dwDesiredAccess
    FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,  // share mode
    NULL,                                                    // default security attributes
    OPEN_EXISTING,                                           // disposition
    // It is always set, no matter whether you explicitly specify it or not. This means, that access
    // must be aligned with sector size so we can only read a number of bytes that is a multiple of the sector size.
    FILE_FLAG_NO_BUFFERING,  // file attributes
    NULL                     // do not copy file attributes
    );

    if (volume == INVALID_HANDLE_VALUE) {
        // An error occurred!
        cout << "Failed to open volume" << endl;
        return nullptr;
    }

    return volume;
}


bool NTFSChangesWatcher::CreateJournal(HANDLE volume) {

    DWORD byte_count;
    CREATE_USN_JOURNAL_DATA create_journal_data;

    bool ok = DeviceIoControl(volume, // handle to volume
        FSCTL_CREATE_USN_JOURNAL,     // dwIoControlCode
        &create_journal_data,         // input buffer
        sizeof(create_journal_data),  // size of input buffer
        NULL,                         // lpOutBuffer
        0,                            // nOutBufferSize
        &byte_count,                  // number of bytes returned
        NULL) != 0;                   // OVERLAPPED structure

    if (!ok) {
        // An error occurred!
    }

    return ok;
}


bool NTFSChangesWatcher::LoadJournal(HANDLE volume, USN_JOURNAL_DATA* journal_data) {

    DWORD byte_count;

    // Try to open journal.
    if (!DeviceIoControl(volume, FSCTL_QUERY_USN_JOURNAL, NULL, 0, journal_data, sizeof(*journal_data), &byte_count,
        NULL)) {

        // If failed (for example, in case journaling is disabled), create journal and retry.

        if (CreateJournal(volume)) {
            return LoadJournal(volume, journal_data);
        }

        return false;
    }

    return true;
}

void NTFSChangesWatcher::WatchChanges() {

    auto u_buffer = make_unique<char[]>(kBufferSize);

    auto read_journal_query = GetWaitForNextUsnQuery(last_usn_);

    while (true) {

        // This function does not return until new USN record created.
        WaitForNextUsn(read_journal_query.get());

        cout << "New entry created in the journal!" << endl;

        auto journal_query = GetReadJournalQuery(read_journal_query->StartUsn);

        DWORD byte_count;
        if (!ReadJournalRecords(journal_query.get(), u_buffer.get(), byte_count)) {
            // An error occurred.
            cout << "Failed to read journal records" << endl;
        }

        last_usn_ = *(USN*)u_buffer.get();
        read_journal_query->StartUsn = last_usn_;

        // If you need here you can:
        // Read and parse Journal records from the buffer.
        // Notify an NTFSChangeObservers about journal changes.
    }
}

bool NTFSChangesWatcher::WaitForNextUsn(PREAD_USN_JOURNAL_DATA read_journal_data) const {

    DWORD bytes_read;
    bool ok = true;

    // This function does not return until new USN record created.
    ok = DeviceIoControl(volume_, FSCTL_READ_USN_JOURNAL, read_journal_data, sizeof(*read_journal_data),
        &read_journal_data->StartUsn, sizeof(read_journal_data->StartUsn), &bytes_read,
        nullptr) != 0;

    return ok;
   }

   unique_ptr<READ_USN_JOURNAL_DATA> NTFSChangesWatcher::GetWaitForNextUsnQuery(USN start_usn) {

    auto query = make_unique<READ_USN_JOURNAL_DATA>();

    query->StartUsn = start_usn;
    query->ReasonMask = 0xFFFFFFFF;     // All bits.
    query->ReturnOnlyOnClose = FALSE;   // All entries.
    query->Timeout = 0;                 // No timeout.
    query->BytesToWaitFor = 1;          // Wait for this.
    query->UsnJournalID = journal_id_;  // The journal.
    query->MinMajorVersion = 2;
    query->MaxMajorVersion = 2;

    return query;
}


bool NTFSChangesWatcher::ReadJournalRecords(PREAD_USN_JOURNAL_DATA journal_query, LPVOID buffer,
    DWORD& byte_count) const {

    return DeviceIoControl(volume_, FSCTL_READ_USN_JOURNAL, journal_query, sizeof(*journal_query), buffer, kBufferSize,
        &byte_count, nullptr) != 0;
}

unique_ptr<READ_USN_JOURNAL_DATA> NTFSChangesWatcher::GetReadJournalQuery(USN low_usn) {

    auto query = make_unique<READ_USN_JOURNAL_DATA>();

    query->StartUsn = low_usn;
    query->ReasonMask = 0xFFFFFFFF;  // All bits.
    query->ReturnOnlyOnClose = FALSE;
    query->Timeout = 0;  // No timeout.
    query->BytesToWaitFor = 0;
    query->UsnJournalID = journal_id_;
    query->MinMajorVersion = 2;
    query->MaxMajorVersion = 2;

    return query;
}

Now you can use it (for example in the main function for testing):

#include "NTFSChangesWatcher.h"

int _tmain(int argc, _TCHAR* argv[])
{
    auto watcher = new NTFSChangesWatcher('z');
    watcher->WatchChanges();
    return 0;
}

And console output should be like this on every change in the filesystem:

This code was slightly reworked to remove unrelated details and is a part of the Indexer++ project. So for more details, you can refer to the original code.