How could I get the response data from sending POST data using the WebBrowser control?

Question

I am wanting to know how I can get the response data after using the WebBrowser control's overload "Navigate" method. Here is an example of what I am doing:

wb.Navigate("https://live.xbox.com/en-US/Friends/List", "", Encoding.ASCII.GetBytes(post), "Content-Type: application/x-www-form-urlencoded\r\n");

This is navigating my page to the first parameter but I have no idea how to just get the response data from calling this. Also the result I want is the response's body. Could somebody please help me. I tried using the WebRequest and WebResponse ways, but they don't allow me to sign the user into the Xbox Live website for some reason, resulting in myself not being able to do anything and get 411 errors. I want to be able to do this with the WebBrowser though. But if I can't, any help would be greatly appreciated.

Answer 1

Try using a System.Net.HttpWebRequest instead of a WebBrowser control:

try
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://website.com/service");
            request.Method = "GET";
            StringBuilder data = new StringBuilder();
            try
            {
                using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
                {
                    StreamReader reader = new StreamReader(response.GetResponseStream());
                    return reader.ReadToEnd();
                }
            }
            catch (WebException ex)
            {
                WebResponse errorResponse = ex.Response;
                using (Stream responseStream = errorResponse.GetResponseStream())
                {
                    StreamReader reader = new StreamReader(responseStream, Encoding.UTF8);
                    String errorText = reader.ReadToEnd();
                }
                throw;
            }
        }
        catch (Exception)
        {
            throw;
        }

This should be pretty self explainatory, but to summarize, the code will create a request to a remote web server using the HTTP GET command. The response can then be parsed and used from the app (console/winform/service etc..)

Hope this helps.

Answer 2

You can use webbrowser control to login using fake web click events etc. (you seem to have done this already, +1)

Then you can transfer the cookies from webbrowser control to webrequest to do more sane stuff. You can also cache cookies for later too

using System;
using System.Net;
using System.Runtime.InteropServices;
using System.Text;

namespace Sample.Helpers {
    public class NativeMethods {
        public static CookieContainer GetUriCookieContainer(Uri uri) {
            CookieContainer cookies = null;
            int datasize = 8192 * 16; // Determine the size of the cookie
            StringBuilder cookieData = new StringBuilder(datasize);
            if (!InternetGetCookieEx(uri.ToString(), null, cookieData, ref datasize, InternetCookieHttponly, IntPtr.Zero)) {
                if (datasize < 0)
                    return null;
                cookieData = new StringBuilder(datasize); // Allocate stringbuilder large enough to hold the cookie
                if (!InternetGetCookieEx(
                    uri.ToString(),
                    null, cookieData,
                    ref datasize,
                    InternetCookieHttponly,
                    IntPtr.Zero))
                    return null;
            }
            if (cookieData.Length > 0) {
                cookies = new CookieContainer();
                cookies.SetCookies(uri, cookieData.ToString().Replace(';', ','));
            }
            return cookies;
        }
        [DllImport("wininet.dll", CharSet = CharSet.Unicode, SetLastError = true, ThrowOnUnmappableChar = true)]
        private static extern bool InternetGetCookieEx(string url, string cookieName, StringBuilder cookieData, ref int size, Int32 dwFlags, IntPtr lpReserved);
        private const Int32 InternetCookieHttponly = 0x2000;

    }
}

Example usage

var webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.CookieContainer = Cookies;
//webRequest.Headers.Add(HeaderCollection); 
webRequest.Timeout = 10000;
//webRequest.UserAgent = UserAgent;
webRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
webRequest.Method = "POST";
byte[] postBytes;
if (postDataRaw != null) {
    postBytes = Encoding.UTF8.GetBytes(postDataRaw);
}
webRequest.ContentLength = postBytes.Length;
using (var dataStream = webRequest.GetRequestStream()) {
    dataStream.Write(postBytes, 0, postBytes.Length);
}
using (var response = webRequest.GetResponse()) {
    var result = ((HttpWebResponse)response).StatusCode;
    using (var responseStream = response.GetResponseStream()) {
        using (var streamReader = new StreamReader(responseStream)) {
            return streamReader.ReadToEnd();
        }
    }
}