Why does my low level windows key hook stop working?

Question

This is a continuation of my original question Why is D3D10SDKLayers.dll loaded during my DX11 game? I am creating a DX11 game and am using a low-level windows key hook to capture Alt+Enter so that I can toggle fullscreen using my own methods instead of having Windows do it automatically, which inevitably causes problems. A description of this process and details are available in the linked question. My problem is that the key hook consistently stops working after the 6th Alt+Enter for some reason. I am not unregistering it myself.

Here is the key hook code:

LRESULT _stdcall MyClass::WindowsKeyHook( s32 nCode, WPARAM wParam, LPARAM lParam ) {
    printf("Key hook called, nCode: %d. ", nCode);
    if( nCode < 0 || nCode != HC_ACTION )  { // do not process message 
        return CallNextHookEx( MyClassVar.GetWindowsKeyHook(), nCode, wParam, lParam );
    }
    printf(" Key hook status ok.\n");

    BOOL bEatKeystroke = FALSE;
    KBDLLHOOKSTRUCT* p = ( KBDLLHOOKSTRUCT* )lParam;
    switch( wParam ) {
        //NOTE: Alt seems to be a system key when it is PRESSED, but a regular key when it is released...
        case WM_SYSKEYDOWN:
            if(p->vkCode == VK_MENU || p->vkCode == VK_LMENU || p->vkCode == VK_RMENU) {
                MyClassVar.SetAltPressed(TRUE);
            }
            if(MyClassVar.IsAltPressed() && p->vkCode == VK_RETURN) {
                bEatKeystroke = TRUE;
                MyClassVar.SetAltEnterUsed(TRUE);
                printf("Alt+Enter used.\n");
            }
            break;
        case WM_SYSKEYUP:
            //NOTE: releasing alt+enter causes a SYSKEYUP message with code 0x13: PAUSE key...
            break;
        case WM_KEYDOWN:
            break;
        case WM_KEYUP: {
            if(p->vkCode == VK_MENU || p->vkCode == VK_LMENU || p->vkCode == VK_RMENU) {
                MyClassVar.SetAltPressed(FALSE);
            }
            bEatKeystroke = ( !MyClassVar.IsShortcutKeysAllowed() &&
                                ( p->vkCode == VK_LWIN || p->vkCode == VK_RWIN ) );
            break;
        }
    }

    if( bEatKeystroke ) {
        return 1;
    }
    else {
        return CallNextHookEx( MyClassVar.GetWindowsKeyHook(), nCode, wParam, lParam );
    }
}

If you need more information, just tell me what is needed. I have no idea why this is happening, so I'm not sure what kind of information I need to provide. The only way to get rid of a key hook besides unregistering it explicitly is if Windows times it out, as far as I know. All MyClassVar methods are inline to be as fast as possible, and Alt+Enter is handled from a separate thread.

Answer 1

The only way to get rid of a key hook besides unregistering it explicitly is if Windows times it out, as far as I know.

That would be correct, and you might be able to confirm it's timing out by increasing the LowLevelHooksTimeout registry key temporarily.

It's also possible, but wildly unlikely that another keyboard hook's gotten there first and is swallowing all the input before your hook is called (which is exactly what your hook tries to do with certain key combinations).

From your comment, it sounds like you want to pick a full-screen resolution when DXGI switches to full-screen for you.

The following is an excerpt from DirectX Graphics Infrastructure (DXGI): Best Practices. This information may or may not help, as will the content I've elided prior to this excerpt which talks about WM_SIZE.

The methodology of the preceding explanation follows a very particular path. DXGI set the full-screen resolution to the desktop resolution by default. Many applications, however, switch to a preferred full-screen resolution. In such a case, DXGI provides IDXGISwapChain::ResizeTarget. This should be called before calling SetFullscreenState. Although these methods can be called in the opposite order (SetFullscreenState first, followed by ResizeTarget), doing so causes an extra WM_SIZE message to be sent to the application. (Doing so can also cause flickering, since DXGI could be forced to perform two mode changes.) After calling SetFullscreenState, it is advisable to call ResizeTarget again with the RefreshRate member zeroed out. This amounts to a no-operation instruction in DXGI, but it can avoid issues with the refresh rate, which are discussed next.

The DXGI Overview has some more information on the matter, that ResizeTarget might not be as helpful as one would hope it would be:

By default, DXGI chooses the output that contains most of the client area of the window. This is the only option available to DXGI when it goes full-screen itself in response to alt-enter.

However, it does mention that that the size is determined by the client area of the window. Perhaps you want to limit the client area, instead of installing a keyboard hook?

Answer 2

Have you tried disabling DXGI's hook?

IDXGISwapChain::MakeWindowAssociation( DXGI_MWA_NO_WINDOW_CHANGES )

---

Also, creating the swapchain with DXGI_SWAP_CHAIN_FLAG_ALLOW_MODE_SWITCH will cause DXGI to find the display mode that most closely matches your window size. If you've already ResizeTarget()ed to 640x480, and there's at least some 640x480 mode available, you should get it. You'll then get a WM_SIZE to 640x480 that is your opportunity to call ResizeBuffers. if ResizeBuffers is called like this, after the mode switch, it will enable page-flipping (the new back-buffers have to be associated with the monitor, or the system can't page-flip directly). If page-flipping is not enabled like this, then all the Present calls will invoke a blt operation that will suck bandwidth you don't need to spend.

Note that you may or may not get 640x480. If the display is rotated, for example, you'll probably get the next biggest mode that will contain 640x480 which will probably be 768x1024. If you want to work well on rotated displays, you can watch out for this and letterbox yourself into the 4x3 aspect ratio you want.

But it's generally best practice to let DXGI choose the display mode, cuz there are many reasons why the desktop mode might be the best one. The user might have connected the puter to a projector that can only do that one mode. Some LCDs will present the 640x480 at native resolution, i.e. a teeny postage stamp in the middle of the monitor. Ideally, you should code your app to be able to look pretty at basically any resolution, but at least at 4x3, 3x4, 6x9 and 9x6 aspect ratios.

Happy coding -Jeff