IIS slow multithreading

Question

We have a .NET application which is calling over OpenRia services on the server (IIS). This web service call is running a heavy calculation, where we are loading over LoadLibrary some DLL's, which we need to solve some linear systems. We need to go over a list of 1000 events. Every single event is a separate calculation and can be run independently from each other. What we are doing is, that we create on a 64-core machine 60 tasks and every task is taking one event => run the calculation => take the next event => run the calculation and so on until the list is empty. As soon the list is empty our calculation is finished.

We have now the strange behaviour that on the first run the calculation seems to run fast, but when we run the same calculation again it's getting slower on every run. If we restart the server the calculation is running fast again.

We have done an analysis with PerfView and we have seen that on the second/third/fourth run the used threads from the IIS worker process are less than at the beginning. On the first run the IIS worker process is using 60 threads (as we have defined) and on the second the process is using less than 60. On every run the actual threads used are less and less. The first run the calculation needs around 3min. The second run we need 6min and the third run we are already around 15min.

What could be the problem? I have tried to use the ThreadPool, but I have the same effect as with the Tasks.

Here is some sample code:

//This part of code is called after the web service call
ConcurrentStack<int> events = new ConcurrentStack<int>();//This is a list of 1000 entries
ParallelOptions options = new ParallelOptions();
int interfacesDone = 0;
Task[] tasks = new Task[options.MaxDegreeOfParallelism];
for (int i = 0; i < options.MaxDegreeOfParallelism; i++)
{
  tasks[i] = Task.Run(() =>
  {
      StartAnalysis(events);
  });
}
Task.WaitAll(tasks);


private void StartAnalysis(ConcurrentStack<int> events)
{
     while (!events.IsEmpty)
     {
         int index;
         if (events.TryPop(out index))
         {
             DoHeavyCalculation();
         }
     }
}

Answer 1

ASP.NET processes requests by using threads from the .NET thread pool. The thread pool maintains a pool of threads that have already incurred the thread initialization costs.

Therefore, these threads are easy to reuse. The .NET thread pool is also self-tuning. It monitors CPU and other resource utilization, and it adds new threads or trims the thread pool size as needed.