MSI installation interrupted by custom action DLL

Question

I have used a Visual Studio setup project to create an MSI. I have edited the MSI in Orca so that a custom action is performed via a DLL when it is first opened. When I run the MSI, msiexec logs the following:

MSI (c) (E4:BC) [15:28:14:453]: Doing action: CustomAction1
Action 15:28:14: CustomAction1. 
Action start 15:28:14: CustomAction1.
MSI (c) (E4:BC) [15:28:14:453]: Note: 1: 2235 2:  3: ExtendedType 4: SELECT `Action`,`Type`,`Source`,`Target`, NULL, `ExtendedType` FROM `CustomAction` WHERE `Action` = 'CustomAction1' 
MSI (c) (E4:BC) [15:28:14:453]: Creating MSIHANDLE (13) of type 790542 for thread 3260
MSI (c) (E4:B4) [15:28:14:453]: Invoking remote custom action. DLL: C:\DOCUME~1\USERNA~1\LOCALS~1\Temp\MSIA3.tmp, Entrypoint: SampleFunction
MSI (c) (E4:B4) [15:28:14:453]: Closing MSIHANDLE (13) of type 790542 for thread 3260
Action ended 15:28:14: CustomAction1. Return value 3.
MSI (c) (E4:BC) [15:28:14:468]: Doing action: FatalErrorForm
Action 15:28:14: FatalErrorForm. 
Action start 15:28:14: FatalErrorForm.
MSI (c) (E4:BC) [15:28:14:468]: Note: 1: 2235 2:  3: ExtendedType 4: SELECT `Action`,`Type`,`Source`,`Target`, NULL, `ExtendedType` FROM `CustomAction` WHERE `Action` = 'FatalErrorForm' 

The installer wizard then displays the error message: The installer was interrupted before MyProduct could be installed. You need to restart the installer to try again.

The custom DLL is written in C++. Here is the source code:

MyCustomAction.cpp:

// MyCustomAction.cpp : Defines the entry point for the DLL application.
#include "stdafx.h"

BOOL APIENTRY DllMain(HANDLE hModule, DWORD  ul_reason_for_call, LPVOID lpReserved)
{
    return TRUE;
}

UINT __stdcall SampleFunction(MSIHANDLE hModule)
{
        //This is the function that is called by the MSI
        //It is empty because I just want to check that it can be called without interrupting the installer, then I will add the actual functionality
}

MyCustomAction.def:

; MyCustomAction.def
;
; defines the exported functions which will be available to the MSI engine

LIBRARY      "MyCustomAction" 
DESCRIPTION  'Custom Action DLL'

EXPORTS
    SampleFunction

I have also referenced msi.lib in the DLL's additional dependencies. Why does the custom action interrupt the installation when I am currently not explicitly telling it do anything? Any help would be appreciated.

UPDATE:

In Orca, the custom action is in the Binary table and is type 1 in the CustomAction table. The custom action is Immediate and takes place after IsolateComponents and before WelcomeForm in the InstallUISequence table.

Answer 1

You should update your code and your post to indicate that you are actually returning ERROR_SUCCESS. Whether it fixes the issue or not, the point is that it is the correct thing to do. If it doesn't return a value the call sequence will fail and get an error.

Your Dll is probably not loading because of missing dependencies. If you put a simple messagebox call in your code you'll at least see if the code is actually starting to run. C++ will require runtime support Dlls that may not be on the system already.

If it turns out that you have dependencies on C++ runtimes then you'll need to install them before running your MSI. That's what the prerequisites choice is for - it generates a setup.exe to install the dependencies and then installs your MSI.