Custom Action Issue with Advanced Installer

Question

I am using Advanced Installer Enterprise to build a .NET application exe package. When attempting to install the package I encountered the fatal error while calling Custom Action from the Attached native .NET DLL.

I tried installing it on a clean virtual machine and it got installed successfully, but on many of the developer machines running 64-bit Windows 11, it prematurely ends the installation process with the error log posted below.

Can someone help by suggesting what could be the possible reason for this failure?

Here is the failure error message from the log file for quick reference: -

`MSI (c) (CC:A4) [20:57:56:271]: Doing action: InstallDlg
Action start 20:57:56: InstallDlg.
MSI (c) (CC:FC) [20:57:56:272]: PROPERTY CHANGE: Deleting ValidLicense property. Its current value is '1.0.0.0'.
MSI (c) (CC:B8) [20:57:58:700]: PROPERTY CHANGE: Adding EDIT_1_PROP_1 property. Its value is 'g'.
MSI (c) (CC:B8) [20:57:58:797]: PROPERTY CHANGE: Modifying EDIT_1_PROP_1 property. Its current value is 'g'. Its new value: 'gs'.
MSI (c) (CC:B8) [20:57:58:882]: PROPERTY CHANGE: Modifying EDIT_1_PROP_1 property. Its current value is 'gs'. Its new value: 'gsd'.
MSI (c) (CC:B8) [20:57:58:980]: PROPERTY CHANGE: Modifying EDIT_1_PROP_1 property. Its current value is 'gsd'. Its new value: 'gsdg'.
MSI (c) (CC:B8) [20:57:59:037]: PROPERTY CHANGE: Modifying EDIT_1_PROP_1 property. Its current value is 'gsdg'. Its new value: 'gsdgd'.
Action ended 20:58:00: InstallDlg. Return value 1.
MSI (c) (CC:A4) [20:58:00:856]: Doing action: AI_DATA_SETTER_2
Action start 20:58:00: AI_DATA_SETTER_2.
MSI (c) (CC:A4) [20:58:00:856]: PROPERTY CHANGE: Adding CustomActionData property. Its value is 'gsdgd1.1.1'.
Action ended 20:58:00: AI_DATA_SETTER_2. Return value 1.
MSI (c) (CC:A4) [20:58:00:856]: Doing action: LicenseDlg
Action start 20:58:00: LicenseDlg.
Action ended 20:58:00: LicenseDlg. Return value 1.
MSI (c) (CC:A4) [20:58:00:857]: Doing action: Validate_License
Action start 20:58:00: Validate_License.
MSI (c) (CC:8C) [20:58:00:909]: Invoking remote custom action. DLL: C:\Users\SUDHIR~1\AppData\Local\Temp\MSIF340.tmp, Entrypoint: ValidateLicense1
SFXCA: Failed to create new CA process via RUNDLL32. Error code: 5
CustomAction Validate_License returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 20:58:00: Validate_License. Return value 3.
MSI (c) (CC:A4) [20:58:00:956]: Doing action: FatalError
Action start 20:58:00: FatalError.
MSI (c) (CC:FC) [20:58:09:423]: PROPERTY CHANGE: Modifying Log_Action property. Its current value is '{}'. Its new value: 'failure'.
Action ended 20:58:09: FatalError. Return value 1.
MSI (c) (CC:A4) [20:58:09:424]: Doing action: AI_DATA_SETTER_3
Action start 20:58:09: AI_DATA_SETTER_3.
MSI (c) (CC:A4) [20:58:09:424]: PROPERTY CHANGE: Modifying CustomActionData property. Its current value is 'gsdgd1.1.1'. Its new value: 'failure1.1.1'.
Action ended 20:58:09: AI_DATA_SETTER_3. Return value 1.
MSI (c) (CC:A4) [20:58:09:425]: Doing action: FatalError
Action start 20:58:09: FatalError.
Action ended 20:58:09: FatalError. Return value 1.
MSI (c) (CC:A4) [20:58:09:427]: Doing action: Log_Actions
Action start 20:58:09: Log_Actions.
MSI (c) (CC:24) [20:58:09:482]: Invoking remote custom action. DLL: C:\Users\SUDHIR~1\AppData\Local\Temp\MSI14C4.tmp, Entrypoint: LogEntries
SFXCA: Failed to create new CA process via RUNDLL32. Error code: 5
CustomAction Log_Actions returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 20:58:09: Log_Actions. Return value 3.
MSI (c) (CC:A4) [20:58:09:526]: Note: 1: 2205 2:  3: Environment
MSI (c) (CC:A4) [20:58:09:526]: Note: 1: 2228 2:  3: Environment 4: SELECT `Environment`,`Component_` FROM `Environment`
MSI (c) (CC:A4) [20:58:09:955]: Destroying RemoteAPI object.
MSI (c) (CC:C4) [20:58:09:955]: Custom Action Manager thread ending.
=== Verbose logging stopped: 29-11-2023  20:58:09 ===`

Have already tried various options related to reviewing .Net framework versions, turning off Windows Defender, turning off antivirus, reviewing various options in the advanced installer wizard, and checking the presence of RUNDLL32 in C:\Windows\System32. But nothing worked so far.

Answer 1

The call to your custom action DLL results in an error code 5 which is ERROR_ACCESS_DENIED.

This can be caused by an inaccessible file (i.e. file(s) you are trying to replace are held open by another process), trying to write to a protected folder, attempting a file or registry operation which requires elevation as a non-elevated user, etc.

Without knowing more about what said custom action DLL is attempting to do, it is hard to say what could cause this. If you provide more information in your question I will update this answer to reflect that.