Threadpool Starvation C# Async Socket Clients

Question

I have written an application that connects out to multiple servers (50+ at the moment). The start up of the application is slow. It appears to be due to ThreadPool starvation because after taking different timings and wireshark traces the problem always seems to resolve itself once the threads are high enough. I made a change and tripled the amount of minimum threads and that has resolved my issue completely. But that works for now but might not scale to 100 clients.

I have used async methods everywhere and have used the System.IO.Pipelines API for the socket processing. During the start up I have

var t = Task.Run(async () => 
{
    var client = new Client(IP, socket);
    try
    {
        await client.StartAsync(CancellationToken);
    }
    catch (Exception e)
    {
        Log.Error(e, "Unhandled exception");
    }
}

This is starting all the client connections at once. I am currently looking through the rest of the App for any potentially blocking methods/long running synchronous that would force the ThreadPool to start a new thread. I am not sure if this would help because the app appears to be slow on start up, before it even gets to any method that could be the cause of the Work Queue pile up.

1. Should a Windows Service written with async APIs and System.IO.Pipelines be able to scale beyond 50+ concurrent client connections?
2. Should I rearchitect the start up to do so in a more controlled fashion? Only allowing so many simulataneous connection attempts. Then needing to write logic to handle the case where connections are stalling out (timeout on TCP connect) so that we don't get stuck behind 5 connections that are going to stall out

Answer 1

Should a Windows Service written with async APIs and System.IO.Pipelines be able to scale beyond 50+ concurrent client connections?

Yes.

Should I rearchitect the start up to do so in a more controlled fashion? Only allowing so many simulataneous connection attempts.

No, that shouldn't matter. If they're talking to different IPs, then you should be able to connect them all simultaneously without issue.

Depending on your runtime and implementation, there are a couple of synchronous aspects to initial connections that may be happening. The first is DNS lookup, which is (sadly) usually synchronous. The second is proxy detection (only applies if you're using an HTTP component like HttpClient and not something lower-than-HTTP like Socket).

Due to your observed behavior (i.e., works fine if you bump the minimum thread count), you're almost definitely getting some synchronous code somewhere in your Client.Client or Client.StartAsync calls.