How to quit/break from windows message loop in console app and deferences from windows desktop app?

Question

I want to break from the windows message loop. Just like C++ how to break message loop in windows hook . I came up with one solution, which works fine for Window Desktop Application, but fails for Console Application. How can this happen?

EDIT: I upload my codes to https://github.com/hellohawaii3/Experiment , clone it and then you can reproduce my problem quickly. Thanks!

1.Why I want to do so?

I am writing a console application. First, users are asked to set some options which determine app's behaviors. Then, the app start the message loop and monitor the input of the keyboard/mouse. Users may want to change the setting, so I want to enable users to press some hotkey, quit from the message loop and go back to the beginning of the application.

If you know any way to implement this function without worrying about breaking from the message loop, please tell me! However, I also want to know why my solution fails for console app and works well for desktop app. Thanks for your help!

2.What have I tried.

I use a bool variable 'recieve_quit'. When certain key is pressed on the keyboard, the variable is setting to True by the hook callback function. For every loop getting message, check the variable 'recieve_quit' first and quit when the variable is False.

3.Result of my experiment

For Console APP, The variable 'recieve_quit' is set correctly when certain key is pressed, however, the message loop continues.

For Windows Desktop APP with GUI, I can quit from the message loop as expected.

4.Experiment settings

I am using VS2019, C++, windows 10 home 1909.

I use dll inject to set hook for my console app.

5.My code

I provide toy example codes here, most of which is generated by Visual Studio automatically, do not bother if you think my codes are too loog.

(a)My console app

Console:

// Console.cpp

#include <iostream>
#include "dll_func.h"
#include <windows.h>

int main()
{
    MSG msg;
    HHOOK hhook_tmp2 = SetWindowsHookEx(WH_KEYBOARD_LL, HandleKeyboardEvent, hDllModule, 0);
    while (recieve_quit == false)
    {
        if (GetMessage(&msg, nullptr, 0, 0)) {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
    }
    MessageBox(NULL, TEXT("APP QUIT"), TEXT(" "), MB_OK);
}

(b)my dll containing hook function

My dll_func.h file, following the doc https://learn.microsoft.com/en-us/cpp/build/walkthrough-creating-and-using-a-dynamic-link-library-cpp?view=msvc-160

#pragma once

#ifdef DLL1_EXPORTS
#define DLLFUNC_API __declspec(dllexport)
#else
#define DLLFUNC_API __declspec(dllimport)
#endif

#include "framework.h"

extern "C" DLLFUNC_API bool recieve_quit;
extern "C" DLLFUNC_API LRESULT CALLBACK HandleKeyboardEvent(int nCode, WPARAM wParam, LPARAM lParam);// refer to https://stackoverflow.com/a/60913531/9758790
extern "C" DLLFUNC_API HMODULE hDllModule;//or whatever name you like 

My dll_func.cpp file, containing the definition of hook function.

#include "pch.h"
#include <windows.h>
#include "dll_func.h"

bool recieve_quit = false;
LRESULT CALLBACK HandleKeyboardEvent(int nCode, WPARAM wParam, LPARAM lParam)
{
    PKBDLLHOOKSTRUCT p = (PKBDLLHOOKSTRUCT)lParam;
    if (wParam == WM_KEYDOWN) {
        if (p->vkCode == VK_F8) {
            recieve_quit = true;
        }
    }

    return CallNextHookEx(NULL, nCode, wParam, lParam);
}

The dllmain.cpp is also modified a little after it is created by Visual Studio as suggested by https://stackoverflow.com/a/60913531/9758790 to get the hinstance of the dll for dll injection.

// dllmain.cpp
#include "pch.h"
#include "dll_func.h"

// refer to https://stackoverflow.com/a/60913531/9758790
HMODULE hDllModule;//or whatever name you like 
BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    hDllModule = hModule;
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

(c) My Desktop application.

I select 'Creating Windows Desktop Application' when created the solution with Visual Studio. Most of the codes is generated automatically by VS. The codes I added and modifed is surrounded by ***** in the codes.

// GUI.cpp
//

#include "framework.h"
#include "GUI.h"

#define MAX_LOADSTRING 100

HINSTANCE hInst;
WCHAR szTitle[MAX_LOADSTRING];
WCHAR szWindowClass[MAX_LOADSTRING];

// *********************** ADDED BY ME！ ***********************
// same as dll_func.cpp
bool recieve_quit = false;
LRESULT CALLBACK HandleKeyboardEvent(int nCode, WPARAM wParam, LPARAM lParam)
{
    //FILE* file;
    //fopen_s(&file, "./temp0210222.txt", "a+");
    //fprintf(file, "Function keyboard_hook called.n");
    //fclose(file);
    PKBDLLHOOKSTRUCT p = (PKBDLLHOOKSTRUCT)lParam;
    if (wParam == WM_KEYDOWN) {
        if (p->vkCode == VK_F8) {
            recieve_quit = true;
        }
    }

    return CallNextHookEx(NULL, nCode, wParam, lParam);
}
// *********************** END OF MY CODES ***********************

ATOM                MyRegisterClass(HINSTANCE hInstance);
BOOL                InitInstance(HINSTANCE, int);
LRESULT CALLBACK    WndProc(HWND, UINT, WPARAM, LPARAM);
INT_PTR CALLBACK    About(HWND, UINT, WPARAM, LPARAM);

int APIENTRY wWinMain(_In_ HINSTANCE hInstance,
                     _In_opt_ HINSTANCE hPrevInstance,
                     _In_ LPWSTR    lpCmdLine,
                     _In_ int       nCmdShow)
{
    UNREFERENCED_PARAMETER(hPrevInstance);
    UNREFERENCED_PARAMETER(lpCmdLine);

    // *********************** ADDED BY ME！ ***********************
    HHOOK hhook_tmp2 = SetWindowsHookEx(WH_KEYBOARD_LL, HandleKeyboardEvent, hInst, 0);
    // *********************** END OF MY CODES ***********************

    LoadStringW(hInstance, IDS_APP_TITLE, szTitle, MAX_LOADSTRING);
    LoadStringW(hInstance, IDC_GUI, szWindowClass, MAX_LOADSTRING);
    MyRegisterClass(hInstance);

    if (!InitInstance (hInstance, nCmdShow))
    {
        return FALSE;
    }

    HACCEL hAccelTable = LoadAccelerators(hInstance, MAKEINTRESOURCE(IDC_GUI));

    MSG msg;

    // main message loop:
    // *********************** MODIFIED BY ME！ ***********************
    //while (GetMessage(&msg, nullptr, 0, 0))
    //{
    //    if (!TranslateAccelerator(msg.hwnd, hAccelTable, &msg))
    //    {
    //        TranslateMessage(&msg);
    //        DispatchMessage(&msg);
    //    }
    //}
    while (recieve_quit == false)
    {
        if (GetMessage(&msg, nullptr, 0, 0)) {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }
    }
    MessageBox(NULL, TEXT("APP QUIT"), TEXT(" "), MB_OK);
    // *********************** END OF　MODIFICATION ***********************

    return (int) msg.wParam;
}



//
//  Function: MyRegisterClass()
ATOM MyRegisterClass(HINSTANCE hInstance)
{
    WNDCLASSEXW wcex;

    wcex.cbSize = sizeof(WNDCLASSEX);

    wcex.style          = CS_HREDRAW | CS_VREDRAW;
    wcex.lpfnWndProc    = WndProc;
    wcex.cbClsExtra     = 0;
    wcex.cbWndExtra     = 0;
    wcex.hInstance      = hInstance;
    wcex.hIcon          = LoadIcon(hInstance, MAKEINTRESOURCE(IDI_GUI));
    wcex.hCursor        = LoadCursor(nullptr, IDC_ARROW);
    wcex.hbrBackground  = (HBRUSH)(COLOR_WINDOW+1);
    wcex.lpszMenuName   = MAKEINTRESOURCEW(IDC_GUI);
    wcex.lpszClassName  = szWindowClass;
    wcex.hIconSm        = LoadIcon(wcex.hInstance, MAKEINTRESOURCE(IDI_SMALL));

    return RegisterClassExW(&wcex);
}

//
//   Function: InitInstance(HINSTANCE, int)
BOOL InitInstance(HINSTANCE hInstance, int nCmdShow)
{
   hInst = hInstance;

   HWND hWnd = CreateWindowW(szWindowClass, szTitle, WS_OVERLAPPEDWINDOW,
      CW_USEDEFAULT, 0, CW_USEDEFAULT, 0, nullptr, nullptr, hInstance, nullptr);

   if (!hWnd)
   {
      return FALSE;
   }

   ShowWindow(hWnd, nCmdShow);
   UpdateWindow(hWnd);

   return TRUE;
}

//
//  Function: WndProc(HWND, UINT, WPARAM, LPARAM)
LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
    switch (message)
    {
    case WM_COMMAND:
        {
            int wmId = LOWORD(wParam);
            switch (wmId)
            {
            case IDM_ABOUT:
                DialogBox(hInst, MAKEINTRESOURCE(IDD_ABOUTBOX), hWnd, About);
                break;
            case IDM_EXIT:
                DestroyWindow(hWnd);
                break;
            default:
                return DefWindowProc(hWnd, message, wParam, lParam);
            }
        }
        break;
    case WM_PAINT:
        {
            PAINTSTRUCT ps;
            HDC hdc = BeginPaint(hWnd, &ps);
            EndPaint(hWnd, &ps);
        }
        break;
    case WM_DESTROY:
        PostQuitMessage(0);
        break;
    default:
        return DefWindowProc(hWnd, message, wParam, lParam);
    }
    return 0;
}

// "About"
INT_PTR CALLBACK About(HWND hDlg, UINT message, WPARAM wParam, LPARAM lParam)
{
    UNREFERENCED_PARAMETER(lParam);
    switch (message)
    {
    case WM_INITDIALOG:
        return (INT_PTR)TRUE;

    case WM_COMMAND:
        if (LOWORD(wParam) == IDOK || LOWORD(wParam) == IDCANCEL)
        {
            EndDialog(hDlg, LOWORD(wParam));
            return (INT_PTR)TRUE;
        }
        break;
    }
    return (INT_PTR)FALSE;
}

(d)Tips for reproducing my results

For Console APP: Build a new console application solution named Console, and copy my codes to replace Console.cpp. Add a new project to this solution, select Create new project -> DLL, name it as Dll1. Create dll_func.h, dll_func.c, and copy my codes in. Do not forget to modify dllmain.cpp, too. Set the path of AdditionalIncludeDirectories, add dll1.lib to AdditionalDependencies. Set the path to dll and lib files too.

For Windows Desktop APP: Build a now Windows Desktop App solution, name it as GUI. Copy my codes in GUI.cpp and simply run it. Press key F8, the app will quit and pop up a message box as expected.

Answer 1

This post explains why console apps dont receive keyboard messages and how they can handle them: does not go inside the Windows GetMessage loop on console application

In your console case execution enters GetMessage and never exits it. The hook receives a notification and correctly sets recieve_quit. Since the execution never exits GetMessage, recieve_quit is not checked.

This answer is about handling getting keypresses in a console app: https://stackoverflow.com/a/6479673/4240951

In general - add a hidden window to your console app or check GetAsyncState of the needed key.

There is no need to set hooks when you have a message loop. You may check keypresses as follows:

while (recieve_quit == false)
{
    if (GetMessage(&msg, nullptr, 0, 0)) {
        if (msg.message == WM_KEYDOWN && msg.wParam == VK_F8)
            recieve_quit = true;
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }
}