C# WebRequest returns intermittent 500 errors from valid URLs

Question

I have an application that was written to read and parse a large quantity of XML from a web service. It's been working fine for months. Quite suddenly one particular section of the code is throwing 500 errors when trying to get responses from the server.

On investigation, hitting one of these problematic URLs in a browser gets a valid response and the expected XML. To further add to the mystery, this section iterates through a number of requests varying the parameters in the URL, and the failures are intermittent - there's not one particular call that's causing the issue.

I have tried hitting the failure, requesting via the browser, then proceeding with the url from the code in debug mode: this still results in a 500 error.

Here's the code - there's some notes on attempted fixes in the comments. Essentially I've seen a number of questions on this already, and the suggested fix is to add a user-agent. It didn't work in this case, and previous questions haven't had intermittent problems.

private static HttpWebResponse RetryGetResponse(WebRequest request)
{
    int failCounter = 1;

    while (failCounter < 20)
    {
        try
        {           
            // I tried this as a fix for the issue
            // the code functions but it didn't fix the bug
            // HttpWebRequest req = (HttpWebRequest)request;
            // req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
            // return (HttpWebResponse)req.GetResponse();

            // this is original code
            return (HttpWebResponse)request.GetResponse();
        }
        catch (WebException ex)
        {
            if (ex.Status != WebExceptionStatus.ReceiveFailure &&
                ex.Status != WebExceptionStatus.ConnectFailure &&
                ex.Status != WebExceptionStatus.KeepAliveFailure)
            {
                LogError(string.Format("Failed to contact server, retrying {0} times", failCounter), ex);
                Thread.Sleep(failCounter * 10000);
                failCounter++;
            }
        }
    }

    throw new Exception("Fatal error - unable to connect");
}

public static void PopulateActions()
{
    // this fetches data to iterate through
    DataTable customers = GetTableData("Customers");

    foreach (DataRow row in customers.Rows)
    {
        string url = string.Format(baseUrlTemplate, "subscribers", "actions.xml?email=" + HttpUtility.UrlEncode(row["EmailAddress"].ToString()));

        WebRequest request = WebRequest.Create(url);
        request.Timeout = 999999;
        request.Credentials = new NetworkCredential(networkKey, "");

        using (WebResponse response = RetryGetResponse(request))
        {
            XDocument doc = XDocument.Load(response.GetResponseStream());
            IEnumerable<XElement> emails = doc.Root.Elements("Email");

            foreach (XElement email in emails)
            {
                // parse the xml and record to Db
            }
        }
    }
}

What else can I do to investigate and attempt to fix this infuriating intermittent problem?

EDIT: Response details

StatusCode: InternalServerError
StatusDescription: "Internal Server Error"

Headers:
{
Vary: Accept-Encoding
X-Cache: MISS
X-Code: 500
X-Error-Type: WS-Failure
X-Request-Duration: 30457ms
X-Request-Id: 979b6ee7-59c0-41e8-8acc-a38ec9b4087b
transfer-encoding: chunked
Connection: keep-alive
Cache-Control: private, s-maxage=0
Content-Type: application/xml
Date: Thu, 05 Jan 2017 11:02:32 GMT
P3P: CP="OTI DSP COR CUR IVD CONi OTPi OUR IND UNI STA PRE"
Server: cm-api-server
}