Random 408 error on POST request

Question

I'm trying to send a POST request to a website in C#, and then parse the html in the response to get certain item names from it. However, I am getting a 408 error about 50% of the time I run the program. Here is my code (most is taken from here: https://msdn.microsoft.com/en-us/library/debx8sh9.aspx):

using System;
using System.IO;
using System.Net;
using System.Text;
using System.Collections.Generic;
using HtmlAgilityPack;
using System.Net;
using System.Diagnostics;
using Newtonsoft.Json.Linq;
using System.Linq;
using Fizzler.Systems.HtmlAgilityPack;

class FinderClass
{
    //some irrelevant code here

    public int getItemIndex(string itemName)
    {
        itemName = itemName.Replace(" ", "+");      //formatting for request
        itemName = itemName.Replace("|", "%7C");


        //taken from https://msdn.microsoft.com/en-us/library/debx8sh9(v=vs.110).aspx :

        WebRequest request = WebRequest.Create("http://csgolounge.com/ajax/tradeCsRightTmp.php");   //address to send request
        request.Method = "POST";
        string postData = "type=Type+-+All&quality=0&exterior=0&fraze=" + itemName + "&search=1&page=1";    //request parameters
        byte[] byteArray = Encoding.UTF8.GetBytes(postData);
        request.ContentType = "application/x-www-form-urlencoded";
        request.ContentLength = byteArray.Length;

        Stream dataStream = request.GetRequestStream();
        dataStream.Write(byteArray, 0, byteArray.Length);
        dataStream.Close();

        WebResponse response = request.GetResponse();
        Debug.WriteLine(((HttpWebResponse)response).StatusDescription);
        dataStream = response.GetResponseStream();

        StreamReader reader = new StreamReader(dataStream);
        string responseFromServer = reader.ReadToEnd();
        Debug.WriteLine(responseFromServer);    //print response to debug console (temporary)

        reader.Close();
        dataStream.Close();
        response.Close();

        return -1;      //placeholder for when item index is parsed from html and returned
    }
}

The exception that is thrown:

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The remote server returned an error: (408) Request Timeout.

Normally I would think there is a straightforward solution to this, but since it only happens half the time I run the program, it might be something more complicated.

Answer 1

Any time that I hear "so and such network problem happens after a while", I tend to look at the code and see if using blocks are being used correctly.

using blocks should be used whenever you

1. Create some resource, and
2. Use that resource, and
3. Finish with that resource in the same method

(there are some other times, but this is the most frequent case)

In your code, you are creating a WebResponse, a Stream, and a StreamReader, you're using them, then you're finishing with them, and you're not cleaning up.

Manually using a Dispose or Close call isn't always enough. These will get skipped over if an exception is thrown in the code and unhandled. A using block will ensure that the cleanup happens, even in the presence of unhandled exceptions.

Answer 2

This error is because of a timeout. On the face of it, it looks as though the server is timing out. So, I'd investigate the server side to see how long the call is taking. However, there's a chance that it may just be a client side problem because I sometimes get this error when I pause the debugger on the client side. I.e. the Request is sent to the server, the server sends the Response back, but the client doesn't accept it. I know this sounds funny, and I haven't been able to figure out why this happens myself. Anyway, you can try setting the timeout to something much larger in the WebRequest on the client side:

request.Timeout = 60000;