DotNet app receives Gateway 502 (proxy server error) - after posting large file to web server

Question

I've seen questions like this asked in a few places but I haven't seen an answer which helps me. Hoping that you can assist!

Situation:

• Uploading files using a .NET workstation client to an Apache web server backed by Tomcat.
• Small files work just fine. Larger files POSTS are rejected with http error 401 around the .copyto command. The subsequent GetResponse errors out with the message "The remote server returned an error: (502) Bad Gateway. The proxy server received an invalid response from an upstream server" The error is seen immediately. It's not timing out.
• CURL can upload the same "large" file without issue.
• The terms "small" and "large" are relative: small = 88 bytes. large = 3.17 MB. I understand that this "large" file is tiny for most purposes.

So, the problem must be in my code. Here is a simplified form of that code (leaving out the exception handlers).

    Dim request As HttpWebRequest = CType(WebRequest.Create("http://mysite"), HttpWebRequest)
    request.Method = "POST"
    request.Credentials = New NetworkCredential("userid","password")
    request.ContentType = "application/octet-stream"
    request.SendChunked = True
    request.UserAgent = "Support Tools"
    request.CookieContainer = New CookieContainer

    Using fileStream As Stream = New FileStream("C:\sourceFile", FileMode.Open)
        request.ContentLength = fileStream.Length
        Dim uploadStream As Stream = request.GetRequestStream
        fileStream.CopyTo(uploadStream)
    End Using

    Dim response As WebResponse = request.GetResponse()

Thank you for your time.

Per request, here is the complete code. Note that it relies upon variables defined elsewhere within the class.

    Public Sub SubmitFileToSis(ByVal operation As String, ByVal sourceFile As String)
    Dim request As HttpWebRequest = WebRequest.Create(_EndPoints(operation.ToLower))
    request.Method = "POST"
    request.Credentials = New NetworkCredential(_SisUserId, _SisUserPassword)
    request.ContentType = "application/octet-stream"
    request.SendChunked = True
    request.UserAgent = "Support  Tools"
    request.CookieContainer = New CookieContainer

    Try
        Using fileStream As Stream = New FileStream(sourceFile, FileMode.Open)
            request.ContentLength = fileStream.Length
            Dim uploadStream As Stream = request.GetRequestStream
            fileStream.CopyTo(uploadStream)
        End Using
    Catch ex As WebException
        If (ex.Status = WebExceptionStatus.ProtocolError) Then
            Dim response As WebResponse = ex.Response
            Using sr As StreamReader = New StreamReader(response.GetResponseStream())
                _SubmissionResult = String.Format("Error Uploading: {0} - {1} - {2}", ex.Status.ToString, ex.Message, sr.ReadToEnd())
            End Using
        Else
            _SubmissionResult = String.Format("Error Uploading: {0} - {1} ", ex.Status.ToString, ex.Message)
        End If
        Exit Sub
    Catch ex As Exception
        _SubmissionResult = String.Format("Error Uploading: {0} ", ex.Message)
        Exit Sub
    End Try

    Try
        Dim response As WebResponse = request.GetResponse()
        Using sr As New StreamReader(response.GetResponseStream(), Encoding.UTF8)
            _SubmissionResult = sr.ReadToEnd()
        End Using
        _SubmittedDataSetUid = GetBetween(_SubmissionResult, "reference code ", " to track")
    Catch ex As WebException When (ex.Status = WebExceptionStatus.ProtocolError)
        If (ex.Status = WebExceptionStatus.ProtocolError) Then
            Dim response As WebResponse = ex.Response
            Using sr As StreamReader = New StreamReader(response.GetResponseStream())
                _SubmissionResult = String.Format("Error Reading Response: {0} - {1} - {2}", ex.Status.ToString, ex.Message, sr.ReadToEnd())
            End Using
        Else
            _SubmissionResult = String.Format("Error Reading Response: {0} - {1} ", ex.Status.ToString, ex.Message)
        End If
        Exit Sub
    Catch ex As Exception
        _SubmissionResult = String.Format("Error Reading Response: {0} ", ex.Message)
        Exit Sub
    End Try

End Sub

Answer 1

Eureka!! I needed to add "Authorization" to the HttpWebRequest Headers. There were a couple of other minor changes but they were for readability and cleanliness.

The new working template (at least for crawling stage) is:

Dim request As HttpWebRequest = WebRequest.Create("http://mysite")
request.Method = WebRequestMethods.Http.Post
request.Credentials = New NetworkCredential("userid","password")
request.ContentType = "application/octet-stream"
request.SendChunked = True
request.UserAgent = "Support Tools"
request.CookieContainer = New CookieContainer
Dim authInfo As String = Convert.ToBase64String(Encoding.UTF8.GetBytes(String.Format("{0}:{1}", "userid","password")))
request.Headers.Add("Authorization", String.Format("Basic {0}", authInfo))

Using fileStream As Stream = New FileStream("C:\sourceFile", FileMode.Open, FileAccess.Read)
    request.ContentLength = fileStream.Length
    Dim uploadStream As Stream = request.GetRequestStream
    fileStream.CopyTo(uploadStream)
End Using

Dim response As WebResponse = request.GetResponse()