DLL built from static lib is flashing a console

Question

Here is a trouble I'm having and struggling with it for some days.

1. We have a software artifact which we distribute both as a DLL and a static library;
2. One client wants to use the static library with a number of other static libraries to build this gigantic DLL (bussiness requirements: just one single DLL file);
3. He can actually build and link the gigantic DLL but when he loads it somehow (using VB6 or Delphi or C#), it flashes a console window during load;
4. The client doesn't like this console flashing (neither do I) and I already tried a number of solutions: adding -Wl,--subsystem,windows and -mwindows to the linker line when generating the DLL (important info: the DLL is built using GCC toolchain on MSYS2), searching for windows APIs that could have the side-effect of displaying a console, trying to remove some dependencies for the same reason;
5. If the client uses our DLL directly (with a number of other DLLs) everything works fines (no console flashing). Both our libs (DLL and static) are built from the very same codebase;
6. I also tried to remove parts of the code and came with the conclusion that it is actually my own code that's making the console flasing. I just don't know where. The DLL is written in a mix of C and C++, uses templates and some windows APIs;
7. As a last try, I searched for some lib that could be actually calling the AllocConsole API but found nothing;
8. This lib is actually a higher level wrapper around a number of well-known libraries: zlib, libpng, cairo, freetype, etc. I wonder if any of these could be the culprit here but I also wonder why in the DLL build it does not show up.

Could someone, please, give me some directions?

Thank you!

Answer 1

`"I just learned that there is a number of fprintf(stderr, calls in several places in the code. Do you think that these maybe the culprit?"'

ANSWER: It could be. Here is some "short-term fix" (or temporary workaround) code to help you find out. Somewhere near the program entry/start add this code to close stderr (and optionally close stdout).

Here is some runnable code so you can see how effective this at shutting down output to stderr (and stdout). I want to emphasize that is for the short-term; just for discovery phase, to see if it helps.

Closing stdout and stderr and then writing to them may lead to undefined behavior(!):

#include <stdio.h>

int main()
{
    fprintf(stdout, "Text for output to stdout\n");
    fprintf(stderr, "Text for output to stderr\n");

    /* Short- term fix: close stderr (and optionally stdout) */
    fclose(stdout);
    fclose(stderr);

    fprintf(stdout, "Text to stdout cannot display!\n");
    fprintf(stderr, "Text to stderr cannot display!\n");

    return 0;
}

Even if the above code doesn't solve the issue, the more capable, longer-term and safer solution would be something like the following. In this solution, we open a logging file using FILE pointer error_logfile. Then we replace all instances of "fprintf(stderr" with "fprintf(error_logfile." This replacement could be done in a single step with a capable editor like Notepad++ or something similar.

This is better because it is standard C. As noted, our previous solution closes stderr but allows writing to it, which may lead to undefined behavior.

Additionally, this preserves (at least for now) the intent of the original code, and that is to capture error messages. FWIW, I'm doing this kind of error logging in about 50% of my code these days. It is not superior to a debug session, but it can be left in code indefinitely, usefully revealing problems even into production, depending on your client/customer:

#include <stdio.h>

int main()
{
    /* Long-term fix:
    * 1.) Add this code block to the code */
    FILE *error_logfile = fopen( "error.log", "w+" );
    if( error_logfile == NULL )
    {
        printf("The debug logging file could not be opened.\n");
        return 0;
    }
    /* 2.) Replace all instances of "fprintf(stderr" with "fprintf(error_logfile" */
    fprintf(error_logfile, "Text for output to Error Log File.\n");
    return 0;
}

Answer 2

I found out the problem. There is this tiny little piece of code that actually needs to enumerate Windows fonts. No problem, just use the system function, enumerate fonts via Windows registry using the reg cmd line utility, redirect to a txt file, process it and voila, fonts enumerated.

And just for the record, don't forget to put it into a separate thread otherwise it will show in the debugger in an easy way and help the poor guy that is mantaining the code (me, btw).

Thank you for all your help.