HttpWebResponse won't scale for concurrent outbound requests

Question

I have an ASP.NET 3.5 server application written in C#. It makes outbound requests to a REST API using HttpWebRequest and HttpWebResponse.

I have setup a test application to send these requests on separate threads (to vaguely mimic concurrency against the server).

Please note this is more of a Mono/Environment question than a code question; so please keep in mind that the code below is not verbatim; just a cut/paste of the functional bits.

Here is some pseudo-code:

// threaded client piece
int numThreads = 1;
ManualResetEvent doneEvent;

using (doneEvent = new ManualResetEvent(false))
        {

            for (int i = 0; i < numThreads; i++)
            {

                ThreadPool.QueueUserWorkItem(new WaitCallback(Test), random_url_to_same_host);

            }
            doneEvent.WaitOne();
        }

void Test(object some_url)
{
    // setup service point here just to show what config settings Im using
    ServicePoint lgsp = ServicePointManager.FindServicePoint(new Uri(some_url.ToString()));

        // set these to optimal for MONO and .NET
        lgsp.Expect100Continue = false;
        lgsp.ConnectionLimit = 100;
        lgsp.UseNagleAlgorithm = true;
        lgsp.MaxIdleTime = 100000;        

    _request = (HttpWebRequest)WebRequest.Create(some_url);


    using (HttpWebResponse _response = (HttpWebResponse)_request.GetResponse())
    {
      // do stuff
    } // releases the response object

    // close out threading stuff

    if (Interlocked.Decrement(ref numThreads) == 0)
    {
        doneEvent.Set();
    }
}

If I run the application on my local development machine (Windows 7) in the Visual Studio web server, I can up the numThreads and receive the same avg response time with minimal variation whether it's 1 "user" or 100.

Publishing and deploying the application to Apache2 on a Mono 2.10.2 environment, the response times scale almost linearly. (i.e, 1 thread = 300ms, 5 thread = 1500ms, 10 threads = 3000ms). This happens regardless of server endpoint (different hostname, different network, etc).

Using IPTRAF (and other network tools), it appears as though the application only opens 1 or 2 ports to route all connections through and the remaining responses have to wait.

We have built a similar PHP application and deployed in Mono with the same requests and the responses scale appropriately.

I have run through every single configuration setting I can think of for Mono and Apache and the ONLY setting that is different between the two environments (at least in code) is that sometimes the ServicePoint SupportsPipelining=false in Mono, while it is true from my machine.

It seems as though the ConnectionLimit (default of 2) is not being changed in Mono for some reason but I am setting it to a higher value both in code and the web.config for the specified host(s).

Either me and my team are overlooking something significant or this is some sort of bug in Mono.

Answer 1

I believe that you're hitting a bottleneck in the HttpWebRequest. The web requests each use a common service point infrastructure within the .NET framework. This appears to be intended to allow requests to the same host to be reused, but in my experience results in two bottlenecks.

First, the service points allow only two concurrent connections to a given host by default in order to be compliant to the HTTP specification. This can be overridden by setting the static property ServicePointManager.DefaultConnectionLimit to a higher value. See this MSDN page for more details. It looks as if you're already addressing this for the individual service point itself, but due to the concurrency locking scheme at the service point level, doing so may be contributing to the bottleneck.

Second, there appears to be an issue with lock granularity in the ServicePoint class itself. If you decompile and look at the source for the lock keyword, you'll find that it uses the instance itself to synchronize and does so in many places. With the service point instance being shared among web requests for a given host, in my experience this tends to bottleneck as more HttpWebRequests are opened and causes it to scale poorly. This second point is mostly personal observation and poking around the source, so take it with a grain of salt; I wouldn't consider it an authoritative source.

Unfortunately, I did not find a reasonable substitute at the time that I was working with it. Now that the ASP.NET Web API has been released, you may wish to give the HttpClient a look. Hope that helps.

Answer 2

If @jake-moshenko is right about ServicePointManager.DefaultConnectionLimit not having any effect if changed in Mono, please file this as a bug in http://bugzilla.xamarin.com/.

However I would try some things before discarding this completely as a Mono issue:

1. Try using the SGen garbage collector instead of the old boehm one, by passing --gc=sgen as a flag to mono.
2. If the above doesn't help, upgrade to Mono 3.2 (which BTW defaults to SGEN GC too), because there has been a lot of fixes since you asked the question.
3. If the above doesn't help, build your own Mono (master branch), as this important pull request about threading has been merged recently.
4. If the above doesn't help, build your own Mono with this pull request added. If it fixes your problem, please add a "+1" to the pull request. It might be a fix for bug 7055.

Answer 3

I know this is pretty old but I'm putting this here in case it might help somebody else who runs into this issue. We ran into the same problem with parallel outbound HTTPS requests. There are a few issues at play.

The first issue is that ServicePointManager.DefaultConnectionLimit did not change the connection limit as far as I can tell. Setting this to 50, creating a new connection, and then checking the connection limit on the service point for the new connection says 2. Setting it on that service point to 50 once appears to work and persist for all connections that will end up going through that service point.

The second issue we ran into was with threading. The current implementation of the mono thread pool appears to create at most 2 new threads per second. This is an eternity if you are doing many parallel requests that start at exactly the same time. To counteract this, we tried setting ThreadPool.SetMinThreads to a higher number. It appears that Mono only creates up to 1 new thread when you make this call, regardless of the delta between the current number of threads and the desired number. We were able to work around this by calling SetMinThreads in a loop until the thread pool had the desired number of idle threads.

I opened a bug about the latter issue because that's the one I'm most confident is not working as intended: https://bugzilla.xamarin.com/show_bug.cgi?id=7055