Disable the on-demand ngen.exe on lightweight windows servers (".NET Framework NGEN v4.0.30319 64")

Question

We're deploying .Net 4 based ASP.net APIs to a large number of Windows servers, and in order to keep AWS costs low, we run them as small as possible.

The problem we're seeing is that after the VM is provisioned and code is deployed, we're seeing high CPU usage for a few minutes. Most of this CPU usage comes from the process ngentask.exe. It's not related to app startup, we're typically seeing this happening a few minutes after the deploy continues and other cpu-heavy processes calm down.

It's also worth mentioning that as part of our deploy process, we touch the app by invoking a web request to a predefined health endpoint in the app, which is simply an empty MVC controller in most of our apps - we do this both to verify that the app is actually running, and to make sure is warmed up and ready to serve traffic.

Looking at the ngen.log, the timestamps match perfectly with CPU usage:

11/13/2018 14:20:13.275 [732]: Task scheduler requested starting 32-bit task
11/13/2018 14:20:13.371 [732]: Task scheduler requested starting 64-bit task
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:\Windows\Microsoft.NET\Framework\v4.0.30319\NGenTask.exe" /RuntimeWide /StopEvent:380
11/13/2018 14:20:13.371 [732]: Launching NGen Task, command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.512 [3104]: NGen Task starting, command line: "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\NGenTask.exe" /RuntimeWide /StopEvent:860
11/13/2018 14:20:13.560 [3104]: Attempting to acquire task lock.

Like I wrote, this can happen as much as 10 minutes after deploy, and by that time the app is already serving requests. This leads me to believe that these tasks are actually not needed - if they were, the app wouldn't function in the 10 minutes between deploy and the completion of these processes.

So my question is: am I safe to disable this task? These are Windows 2016 Core servers only running a single ASP.net web app, so there's no random GUI apps getting installed or anything.

Answer 1

Instead of trying to disable Ngen, you could manually execute it as the final step in your deployment. Ngen normally runs in the background, and is designed to optimise application startup - but ironically it seems to destroy performance for several minutes in the process. Ngen will supposedly auto start only when the server is idle - but once it starts it seems to hog the CPU for several minutes and ive seen that cause issues on smaller servers (specifically AWS Micro instances).

Below one-liner finds the newest version of Ngen and forces an update. It takes a couple of minutes to run, when its done the server should quieten down and behave.

& ([String] $ngenExe = ((Get-ChildItem -Path C:\Windows\Microsoft.NET\Framework64\v*\ngen.exe) | sort -Descending | Select -First 1)) update /force

See: https://learn.microsoft.com/en-us/dotnet/framework/tools/ngen-exe-native-image-generator