Unknown reason for Timeout on HTTP HEAD request

Question

I'm using ASP.NET 3.5 to build a website. One area of the website shows 28 video thumbnail images, which are jpeg's hosted on another webserver. If one or more of these jpegs do not exist, I want to display a locally hosted default image to the user, rather than a broken image link in the browser.

The approach I have taken to implement this is whenever the page is rendered it will perform an HTTP HEAD request to each of the images. If I get a 200 OK status code back, then the image is good and I can write out <img src="http://media.server.com/media/123456789.jpg" />. If I get a 404 Not Found, then I write out <img src="/images/defaultthumb.jpg" />.

If course I don't want to do this every time for all requests, and so I've implemented a list of cached image status objects stored at application level so that each image is only checked once every 5 minutes across all users, but this doesn't really have any bearing on my issue.

This seems to work very well. My problem is that for specific images, the HTTP HEAD request fails with Request Timed Out.

I have set my timeout value very low to only 200ms so that is doesn't delay the page rendering too much. This timeout seems to be fine for most of the images, and I've tried playing around and increasing this during debugging, but it makes no difference even if it's 10s or more.

I write out a log file to see whats happening, and this is what I get (edited for clarify and anonymity):

14:24:56.799|DEBUG|[HTTP HEAD CHECK OK [http://media.server.com/adpm/505C3080-EB4F-6CAE-60F8-B97F77A43A47/videothumb.jpg]]
14:24:57.356|DEBUG|[HTTP HEAD CHECK OK [http://media.server.com/adpm/66E2C916-EEB1-21D9-E7CB-08307CEF0C10/videothumb.jpg]]
14:24:57.914|DEBUG|[HTTP HEAD CHECK OK [http://media.server.com/adpm/905C3D99-C530-46D1-6B2B-63812680A884/videothumb.jpg]]
...
14:24:58.470|DEBUG|[HTTP HEAD CHECK OK [http://media.server.com/adpm/1CE0B04D-114A-911F-3833-D9E66FDF671F/videothumb.jpg]]
14:24:59.027|DEBUG|[HTTP HEAD CHECK OK [http://media.server.com/adpm/C3D7B5D7-85F2-BF12-E32E-368C1CB45F93/videothumb.jpg]]
14:25:11.852|ERROR|[HTTP HEAD CHECK ERROR [http://media.server.com/adpm/BED71AD0-2FA5-EA54-0B03-03D139E9242E/videothumb.jpg]] The operation has timed out
Source: System
Target Site: System.Net.WebResponse GetResponse()
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at MyProject.ApplicationCacheManager.ImageExists(String ImageURL, Boolean UseCache) in d:\Development\MyProject\trunk\src\Web\App_Code\Common\ApplicationCacheManager.cs:line 62

14:25:12.565|ERROR|[HTTP HEAD CHECK ERROR [http://media.server.com/adpm/92399E61-81A6-E7B3-4562-21793D193528/videothumb.jpg]] The operation has timed out
Source: System
Target Site: System.Net.WebResponse GetResponse()
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at MyProject.ApplicationCacheManager.ImageExists(String ImageURL, Boolean UseCache) in d:\Development\MyProject\trunk\src\Web\App_Code\Common\ApplicationCacheManager.cs:line 62

14:25:13.282|ERROR|[HTTP HEAD CHECK ERROR [http://media.server.com/adpm/7728C3B6-69C8-EFAA-FC9F-DAE70E1439F9/videothumb.jpg]] The operation has timed out
Source: System
Target Site: System.Net.WebResponse GetResponse()
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at MyProject.ApplicationCacheManager.ImageExists(String ImageURL, Boolean UseCache) in d:\Development\MyProject\trunk\src\Web\App_Code\Common\ApplicationCacheManager.cs:line 62

As you can see, the first 25 HEAD requests work, and the final 3 do not. It's always the last three.

If I paste one of the failed HEAD request URLs into a web browser: http://media.server.com/adpm/BED71AD0-2FA5-EA54-0B03-03D139E9242E/videothumb.jpg, it loads the image with no problems.

To try to work out what is happening here, I used Wireshark to capture all of the HTTP requests that are sent to the webserver hosting the images. For the log example I've given, I can see 25 HEAD requests for the 25 that were successful, but the 3 that failed do NOT appear in the wireshark trace.

Other than the images having different visual content, there is no difference from one image to the next.

To eliminate any problems with the URL itself (even though it works in a browser) I changed the order by switching one of the first images with one of the last failed three. When I do this, the problem goes away for the one that used to fail, and starts failing for the one that was bumped down to the end of the list.

---

So I think I can deduce from the above that when more than 25 HEAD requests occur in quick succession, subsequent HEAD requests fail regardless of the specific URL. I also know that the issue is on the IIS server rather than the remote image hosting server, due to the lack of requests in the Wireshark trace beyond the first 25.

The code snippet I'm using to perform the HEAD requests is shown below. Can anyone give me any suggestions as to what might be the problem? I've tried various different combinations of request header values, but none of them seem to make any difference. My gut feeling is there is some IIS setting somewhere that limits the number of concurrent HttpWebRequests's to 25 in any one request to an ASP.NET page.

        try {
            HttpWebRequest hwr = (HttpWebRequest)WebRequest.Create(ImageURL);
            hwr.Method = "HEAD";
            hwr.KeepAlive = false;
            hwr.AllowAutoRedirect = false;
            hwr.Accept = "image/jpeg";
            hwr.Timeout = 200;
            hwr.CachePolicy = new System.Net.Cache.RequestCachePolicy(System.Net.Cache.RequestCacheLevel.Reload);
            //hwr.Connection = "close";

            HttpWebResponse hwr_result = (HttpWebResponse)hwr.GetResponse();
            if (hwr_result.StatusCode == HttpStatusCode.OK) {

                Diagnostics.Diags.Debug("HTTP HEAD CHECK OK [" + ImageURL + "]", HttpContext.Current.Request);

                // EXISTENCE CONFIRMED - ADD TO CACHE
                if (UseCache) {
                    _ImageExists.Value.RemoveAll(ie => ie.ImageURL == ImageURL);
                    _ImageExists.Value.Add(new ImageExistenceCheck() { ImageURL = ImageURL, Found = true, CacheExpiry = DateTime.Now.AddMinutes(5) });
                }

                // RETURN TRUE
                return true;

            } else if (hwr_result.StatusCode == HttpStatusCode.NotFound) {
                throw new WebException("404");
            } else {
                throw new WebException("ERROR");
            }            

        } catch (WebException ex) {
            if (ex.Message.Contains("404")) {

                Diagnostics.Diags.Debug("HTTP HEAD CHECK NOT FOUND [" + ImageURL + "]", HttpContext.Current.Request);

                // NON-EXISTENCE CONFIRMED - ADD TO CACHE
                if (UseCache) {
                    _ImageExists.Value.RemoveAll(ie => ie.ImageURL == ImageURL);
                    _ImageExists.Value.Add(new ImageExistenceCheck() { ImageURL = ImageURL, Found = false, CacheExpiry = DateTime.Now.AddMinutes(5) });
                }

                return false;

            } else {

                Diagnostics.Diags.Error(HttpContext.Current.Request, "HTTP HEAD CHECK ERROR [" + ImageURL + "]", ex);

                // ASSUME IMAGE IS OK
                return true;
            }

        } catch (Exception ex) {
            Diagnostics.Diags.Error(HttpContext.Current.Request, "GENERAL CHECK ERROR [" + ImageURL + "]", ex);

            // ASSUME IMAGE IS OK
            return true;
        }

Answer 1

I have solved this myself. The problem was indeed the number of allowed connections, which was set to 24 by default.

In my case, I am going to only perform the image check if the MyHttpWebRequest.ServicePoint.CurrentConnections is less than 10.

To increase the max limit, just set ServicePointManager.DefaultConnectionLimit to the number of concurrent connections you require.

An alternative which may help some people would be to reduce the idle time that is the time a connection waits until it destroys itself. To change this, you need to set MyHttpWebRequest.ServicePoint.MaxIdleTime to the timeout value in milliseconds.