What to do when a DLL needs a bigger stack size than the main exe application?

Question

I have a simulation engine written in Visual C++ 2010 and I'm implementing a DLL plugin based on it for another 3rd party application.

However, I get a stack overflow error when my library is called by the 3rd party app's thread. After debugging it turned out that the DLL requires a bigger stack size than the thread has. Is it possible to extend the current thread's stack size somehow?

I know I should probably review the simulation engine's code and move big objects to the heap. The problem is that the engine is maintained by another vendor and I'd like to avoid modifying their code if possible.

I'm thinking about creating my own thread in the DLL with a bigger stack size and returning the results to the calling thread when the calculation finishes. Is it the right approach?

Thanks, Michal

Answer 1

I'm going to suggest that the first thing you check is how much stack space you're actually using. Are you allocating large objects on the stack? Does your program utilize significant recursion depths? Write a test hook application you can link into your DLL and check how much stack space you use.

If you're allocating large objects on the stack I would strongly suggest moving them to the heap.

If you're doing significant recursion you may wish to investigate using a loop with a heap-based stack maintained by your DLL rather than relying on the application's stack.

I make these suggestions simply because it's much less intrusive for a plugin to be self-contained and not require special stack configuration (or even an entirely separaate thread just to have a larger stack).

Answer 2

Visual Studio gives you two options to change default stack size (1MB):

/F for compiler: https://learn.microsoft.com/cpp/build/reference/f-set-stack-size

/STACK for linker: https://learn.microsoft.com/cpp/build/reference/stack

Answer 3

I had your exact same problem and solved it by spawning a thread with a larger stack size. My code that needs the large stack is well-isolated set-up code, so it was a fairly simple solution for me.

Answer 4

You can check and increase reserved stack size of the main executable. Just open the Command Prompt for VS (I have VS 2019 installed and it is called "Developer Command Prompt for VS 2019"), navigate to the main application's folder and type:

editbin /stack:N main.exe

where N - stack size in bytes (e.g. if you want 4MB type 4194304).

Worked for me.