DDE: Implementing an application which launches correctly via Shell and ddeexec

Question

I am trying to implement an application which registers itself as a DDE Server so that it responds correctly to our custom ".qsx" filetype, just as WinWord responds to the ".docx" filetype.

i.e.

1. If the application is already running and someone double-clicks on a ".qsx" file then we want the already open application to receive the DDE open command and to open the relevant file.
2. However, if the application is not already running then the shell should launch our application, and then establish a DDE connection with the application and tell it to open the file as before.

We have part 1 working flawlessly.

However, for part 2, the Shell launches the command, but then fails with "There was a problem sending the command to the program". This corresponds to return code "SE_ERR_DDEFAIL" (29) returned from the "ShellExecute" Windows API function.

In fact, I have written a custom console application called "ShellExecute.exe" that can reproduce this problem exactly independently of explorer.exe.

If I run ShellExecute.exe and our "DDE server" app is already up, it works correctly. If I run ShellExecute.exe for a ".docx" document and WinWord is not running, WinWord launches correctly and loads the document If I run ShellExecute.exe for our ".qsx" document and our app is not running, ShellExecuteEx launches the application but immediately returns with SE_ERR_DDEFAIL before our server app has even had a chance to register as a DDE server.

For some reason then it seems that for WinWord, ShellExecute waits for it to start up.

I've tried running both cases with procmon.exe running to see what ShellExecute might be looking for in the registry that differentiates our case from WinWord.exe, but I can't find anything.

What I really need is the source code of the ShellExecute algorithm so that I can find out why it is working for WinWord but not for our custom extension.

Can anyone explain, in precise detail, what ShellExecute does to launch an application and how in particular it knows to "wait" for the application to register as a DDE server?

Answer 1

Applications running on XP+ should use IDropTarget to avoid the problems with DDE when there are hung windows.

When using DDE the shell assumes your DDE server is up and running before you start your message loop...

Answer 2

Please don't use DDE. It is a ridiculously obsolete way of doing that that causes all kinds of problems and hangs and depends on the registry.

The modern way to do this is with a global mutex. If you search Stack Overflow you'll find dozens of people asking the same question (maybe start here)