KiUserExceptionDispatch calls into handler in unloaded NativeAot dll

Question

I have a program that loads a NativeAot compiled dll into a process, I was able to unload the module with some hacky approach. However, I recently discovered a problem, that is, the Window exception dispatcher calls into the handler in the NativeAot module even after it was unloaded, no matter where the exception is thrown, causing access violation.

Pseudo code:

HMODULE module = LoadLibraryA("Aot.dll");

// Code that terminates .NET runtime thread and unload dll
.......

// Throw and catch an exception
    try {
        throw exception("argh"); // Access violation executing location 0x00007FF97C8E69B0.
    }
    catch(exception ex){
        cout << "Caught" << endl; // Handler never called
    }

Stack trace from visual sutdio:

    00007ff97a3a69b0() -> This is a function in the unmapped module
    ntdll.dll!RtlpCallVectoredHandlers()
    ntdll.dll!RtlDispatchException()
    ntdll.dll!KiUserExceptionDispatch()
    main()

Any idea on why this is happening and what solution/hack I can use will be appreciated!

Answer 1

As Raymond Chen pointed out in the comments (much appreciated), .NET registeres a vectored exception handler with AddVectoredExceptionHandler.

I hooked this api and check the module of the handler with GetModuleHandleEx during unload, then unregisters it. Now the problem is gone.

There's a link to a demo in the answer of the question linked above.