DEVHIDE
Login Or Sign up

HttpClient / RestSharp, unable to retrieve complete web content

84 Views Asked by At

I am trying to simulate a web site access via C# code. The flow is

  1. HTTP Get the the login Page. This succeeds.
  2. HTTP Post to login page. This returns status 302, I disabled auto redirect in HttpClientHandler. Validated the cookie returned, it has the login cookie.
  3. HTTP Get the actual content page. Returns success code 200, but the content is always trimmed. This is the same page to which step 2 re-directs.

I have tried by even letting the auto-redirect enabled in the HttpClientHandler. Even then the response is trimmed.

In postman, when I directly do step 2 allowing re-direct. The content comes properly.

This used to work sometime back on the website. It's PHP based website and it's protected by CloudFare. Not sure if the CloudFare is recent thing.

I checked the headers sent via the browser for the same and replicated the same in the code but still it doesn't seems to work.

Chrome request headers for step1:

GET /auth-login.php HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-IN,en-US;q=0.9,en-GB;q=0.8,en;q=0.7
Connection: keep-alive
Host: abc.MaskingItForStackOverflow.com
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

Response in chrome for step1:

HTTP/1.1 200 OK
Date: Fri, 23 Feb 2024 16:09:11 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
set-cookie: PHPSESSID=bc4031a1c38246e2fq15ed5f923adead; path=/; secure
expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
pragma: no-cache
vary: Accept-Encoding
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2FxKl8avw4W8R1TOxO3gXyCkYTMaptuI5Z6S2%2FQVQXPy08ozsRJaXJdZLw5E4yjjc3tb%2FBJXTTiBu%2BonGRHks%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 85a0bb524b8b7fdd-MAA
Content-Encoding: gzip
Via: HTTP/1.1 m_proxy_che2
Via: HTTP/1.1 s_proxy_che2

Chrome request headers for step2:

POST /auth-login.php HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-IN,en-US;q=0.9,en-GB;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 242
Content-Type: application/x-www-form-urlencoded
Cookie: _ga=GA1.1.29959955.1687969275; _ga_RRP4199HWQ=deleted; _gcl_au=1.1.454715161.1704898789; PHPSESSID=bc4031a1c38246e2fq15ed5f923adead; _ga_RRP4199HWQ=GS1.1.1708704553.366.1.1708704554.59.0.0
Host: abc.MaskingItForStackOverflow.com
Origin: https://abc.MaskingItForStackOverflow.com
Referer: https://abc.MaskingItForStackOverflow.com/auth-login.php
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

Response in chrome for step2:

HTTP/1.1 302 Found
Date: Fri, 23 Feb 2024 16:10:53 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, no-store, must-revalidate, max-age=0
pragma: no-cache
set-cookie: login_token=KxwmtWKgz0fVQ3XnyzzhZM55FBf4tkS; expires=Sun, 24-Mar-2024 16:10:53 GMT; Max-Age=2592000; secure
set-cookie: username=Ashutosh; expires=Sat, 24-Feb-2024 16:10:53 GMT; Max-Age=86400; secure
set-cookie: usertype=User; expires=Sat, 24-Feb-2024 16:10:53 GMT; Max-Age=86400; secure
location: product-details.php
vary: Accept-Encoding
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2Bb%2FHzNz2UxJ38lDupSsqNS8DG%2F9nH%2BX317G1TIQVxkbUmIYVd%2FAKjQ2CB7DE9iV2KqgjBf1ZveEl7%2BqcY0Z8YQ2R27XlLtnlHvmh4O2zksETAWpkMvMvdY%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 85a0bdca18587fdd-MAA
Via: HTTP/1.1 m_proxy_che2
Via: HTTP/1.1 s_proxy_che2

Chrome request headers for step3:

GET /product-details.php HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-IN,en-US;q=0.9,en-GB;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Cookie: _ga=GA1.1.29959955.1687969275; _ga_RRP4199HWQ=deleted; _gcl_au=1.1.454715161.1704898789; PHPSESSID=bc4031a1c38246e2fq15ed5f923adead; _ga_RRP4199HWQ=GS1.1.1708704553.366.1.1708704554.59.0.0; login_token=KxwmtWKgz0fVQ3XnyzzhZM55FBf4tkS; username=Ashutosh; usertype=User
Host: abc.MaskingItForStackOverflow.com
Referer: https://abc.MaskingItForStackOverflow.com/auth-login.php
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

Response in chrome for step3:

HTTP/1.1 200 OK
Date: Fri, 23 Feb 2024 16:10:55 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
pragma: no-cache
vary: Accept-Encoding
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2BmCa4NokFxiYM2p2ZqgYJeGLq%2FqdmVipkLNz3ro%2BBSoY3l%2FcE0DadqLomhKykmmuUGGhvfaHtcp2hpN0wzJ6xIvlAuZs2VotuVg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 85a0bdd66a7d7ee2-MAA
Content-Encoding: gzip
Via: HTTP/1.1 m_proxy_che2
Via: HTTP/1.1 s_proxy_che2

From the code I set these headers for step 1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-IN,en-US;q=0.9,en-GB;q=0.8,en;q=0.7
Connection: keep-alive
Host: abc.MaskingItForStackOverflow.com
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

From the code I set these headers for step 2

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-IN,en-US;q=0.9,en-GB;q=0.8,en;q=0.7
Cache-Control: max-age=0
Host: abc.MaskingItForStackOverflow.com
Origin: https://abc.MaskingItForStackOverflow.com
Referer: https://abc.MaskingItForStackOverflow.com/auth-login.php
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

From the code I set these headers for step 3

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Language: en-IN,en-US;q=0.9,en-GB;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Host: abc.MaskingItForStackOverflow.com
Referer: https://abc.MaskingItForStackOverflow.com/auth-login.php
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36
sec-ch-ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"

The response header in code via HttpClient is as below:

    {StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Transfer-Encoding: chunked
Connection: keep-alive
pragma: no-cache
vary: Accept-Encoding
x-turbo-charged-by: LiteSpeed
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2FygAzdvahT%2BLZRFN%2FqyPoa5sEwVhapX0RvMwgRhImHcb7v9D%2BvHZUoaNKdBk71QKedjbv2F1I9w6DouYNuurOSsqldc0Mx12d4b3rcyTfHDxU%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
CF-RAY: 85a11083296bb2af-MAA
Cache-Control: no-store, must-revalidate, no-cache
Date: Fri, 23 Feb 2024 17:07:20 GMT
Server: cloudflare
Via: HTTP/1.1 m_proxy_che2
Via: HTTP/1.1 s_proxy_che2
Content-Type: text/html; charset=UTF-8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
}}

But this response is truncated. I have enabled automatic de-compression of the data.

Any idea what might be missing.

Interestingly, when posting to login page via Postman without explicitly adding any other header, the login process works and retrieves the re-directed page.

My C# code:

static Uri baseAddress = new Uri("https://abc.MaskingItForStackOverflow.com");
    static HttpClient CreateClient(CookieContainer cookieContainer)
    {
        HttpClientHandler handler = new HttpClientHandler
        {
            CookieContainer = cookieContainer,
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
        };
        HttpClient client = new HttpClient(handler) { BaseAddress = baseAddress };
        //handler.AllowAutoRedirect = false;
        return client;
    }
    
    static void Main()
    {
        CookieContainer cookieContainer = new CookieContainer();
        HttpClient client = CreateClient(cookieContainer);

        AddHeaders(client, @"WebGet.txt");
        var response = GetRequestHelper(client, "/auth-login.php").Result;
        ReadAndDumpResponse(response);

        //Post login.
        var loginContent = new FormUrlEncodedContent(new[]
        {
            new KeyValuePair<string, string>("username", "***MASKED****"),
            new KeyValuePair<string, string>("deviceid", ""),
            new KeyValuePair<string, string>("password", "***MASKED****"),
            new KeyValuePair<string, string>("btnlogin", "Login"),
        });
        AddHeaders(client, @"WebPost.txt");
        response = PostRequestHelper(client, loginContent, "/auth-login.php").Result;
        ReadAndDumpResponse(response);

        AddHeaders(client, @"WebGet2.txt");
        response = GetRequestHelper(client, "/product-details.php").Result;
        ReadAndDumpResponse(response);

    }
    private static void ReadAndDumpResponse(HttpResponseMessage response)
    {
        Task<string> responseStr = response.Content.ReadAsStringAsync();
        responseStr.Wait();
        string resultValues = responseStr.Result;
        Console.WriteLine(resultValues);
    }
    static void AddHeaders(HttpClient client, string headerFiles)
    {
        var webHeaders = ReadHeaders(headerFiles);
        foreach (var header in webHeaders)
        {
            client.DefaultRequestHeaders.TryAddWithoutValidation(header.Key, header.Value);
        }
    }
    private static Dictionary<string, string> ReadHeaders(string headerFileName)
    {
        Dictionary<string, string> result = new Dictionary<string, string>();

        var lines = File.ReadAllLines(headerFileName);

        foreach (var line in lines)
        {
            if (String.IsNullOrWhiteSpace(line)) continue;
            var colonPos = line.IndexOf(':');
            string key = line.Substring(0, colonPos);
            string value = line.Substring(colonPos + 1);
            result[key] = value;
        }
        return result;
    }

    protected static async Task<HttpResponseMessage> GetRequestHelper(HttpClient httpClient, string url)
    {
        return await httpClient.GetAsync(url).ConfigureAwait(false);
    }
    protected static async Task<HttpResponseMessage> PostRequestHelper(HttpClient httpClient, FormUrlEncodedContent content, string url)
    {
        return await httpClient.PostAsync(url, content);
    }

I tried similar code even in RestSharp and there also the truncation happens exactly at the same point.

c# dotnet-httpclient restsharp
Original Q&A
0

There are 0 best solutions below

Related Questions in C#

Related Questions in DOTNET-HTTPCLIENT

Related Questions in RESTSHARP

Trending Questions

Popular # Hahtags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs

Popular Questions

.

Copyright © 2021 Jogjafile Inc.