Executing more than 1000 HTTP request tasks fails

Question

I'm trying to load test an API. I'm executing a tasks at the same time, each executing an HTTP request. I use Task.WhenAll(mytasks) for waiting for all tasks to be finished. The requests look as follows:

using (var response = await client.SendAsync(request).ConfigureAwait(false))
{
    using (var jsonResponse = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
    {
        var jsonSerializer = new DataContractJsonSerializer(typeof(Borders));
        var borders = (Borders)jsonSerializer.ReadObject(jsonResponse);
        return borders;
    }
}

This works fine up to at least thousand tasks. However, if I start more then a few thousand tasks I run into HttpRequestExceptions:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at BusinessLogic.<GetBorder>d__6d.MoveNext() in c:\BusinessLogic.cs:line 757

So my questions: Why does this happen (with more than 1000 tasks)? How can I prevent this? I could obviously cut my block of tasks into chunks <1000 but I would like to leave this to the underlying system...

Answer 1

I would like to leave this to the underlying system...

That's not a good idea. The .NET Framework has zero competency at determining the optimal degree of parallelism for IO.

It is usually not a good idea to issue that many requests in parallel because the resources that are stressed here are likely maxed out before that. Your backend server is not made to handle that degree of parallelism, apparently. It forcibly cuts you off according to the message.

Just because we have easy async IO with await now does not mean you can spam your resources with 1000 parallel requests.

Use one of the common solutions to perform a series of async actions with a set degree of parallelism. I like http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx but there are solutions based on ActionBlock as well.

Answer 2

I wrote this gist some time ago

//PM> Install-Package Rx-Linq
readonly List<string> _list = new List<string> { "http://www.google.com", "https://www.gmail.com", "http://www.aripaev.ee" };
private readonly string format = "[{0}] {1} : {2} [{3}]";
[Category("WebClient")]
[TestCase("sync" )]
public void SynchronousTest(string name)
{
    DateTime start = DateTime.Now;
    var dict = _list.ToDictionary(o => o, o => new WebClient().DownloadString(new Uri(o)));
    dict.Keys.ToList().ForEach(o => Console.WriteLine(format, DateTime.Now - start, o, dict[o].Length, name));
}
[Category("WebClient")]
[TestCase("async")]
public void AsynchronousTest(string name)
{
    DateTime start = DateTime.Now;
    var dict = _list.ToDictionary(o => o, async o => await new WebClient().DownloadStringTaskAsync(new Uri(o)));
    dict.Keys.ToList().ForEach(o => Console.WriteLine(format, DateTime.Now - start, o, dict[o].Result.Length, name));
}
[Category("WebClient")]
[TestCase("lazy")]
public void LazyTest(string name)
{
    var start = DateTime.Now;
    var dict = _list.ToDictionary(o => o, o => new Lazy<string>(() => new WebClient().DownloadString(new Uri(o))));
    dict.Keys.ToList().ForEach(o => Console.WriteLine(format, DateTime.Now - start, o, dict[o].Value.Length, name));
}

With following results :

[00:00:00.9520952] http://www.google.com : 51348 [sync]
[00:00:00.9520952] https://www.gmail.com : 58704 [sync]
[00:00:00.9520952] http://www.aripaev.ee : 208324 [sync]

[00:00:00.0010001] http://www.google.com : 51412 [lazy]
[00:00:00.6550655] https://www.gmail.com : 58664 [lazy]
[00:00:00.7690769] http://www.aripaev.ee : 208324 [lazy]

[00:00:00.1430143] http://www.google.com : 51366 [async]
[00:00:00.3430343] https://www.gmail.com : 58616 [async]
[00:00:00.5150515] http://www.aripaev.ee : 208324 [async]

Single machine/server can handle 300-500 simultaneous requests, but even that is a stress test to system/network resources.

Answer 3

As usr already stated in his answer, your server is closing the connections, resulting in the error message:

An existing connection was forcibly closed by the remote host.

You still don't know what exactly your bottleneck is. It could be that your server cannot handle the requests, it could be the server blocking them due to some rate limiting feature, and it could also be the stress client itself due to not being able to process the responses quickly enough and therefore keeping too many connections open. You will need to gather additional information before being able to optimize your service.

Instead of trying to write your own stress testing tool, I'd recommend using an established one, like Apache JMeter. You should be able to build a test plan that resembles the test case you want to run.

At some point you will reach a user count a single machine is not able to simulate. There are services (like Redline13) that let you run your tests from EC2 instances and provide you with the tools to analyze the resulting data. You could also use this JMeter EC2 script to execute your test plan from multiple machines.