Keep getting stuck loading when ScrapySharp NavigateToPage

Question

My browser just keeps loading when navigatetopage using scrapysharp and won't go to the next line of code. Below is my code using c# asp.net web form. May I know why? The link I use is working and can manually browse. The code just gets stuck at the Browser.NavigateToPage(new Uri("http://www.asnb.com.my/v3_/asnbv2_0index.php")); and keep loading in the browser. And I am using asp.net webform.

ScrapingBrowser Browser = new ScrapingBrowser();
Browser.AllowAutoRedirect = true; 
Browser.AllowMetaRedirect = true;

WebPage PageResult = Browser.NavigateToPage(new Uri("http://www.asnb.com.my/v3_/asnbv2_0index.php"));
HtmlNode TitleNode = PageResult.Html.CssSelect(".navbar-brand").First();

Answer 1

I was having the same problem and decided not to use Browser.NavigateToPage and instead get the PageResult.Htmlusing an HtmlDocument.

For example:

HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("http://www.asnb.com.my/v3_/asnbv2_0index.php");
HtmlNode TitleNode = doc.DocumentNode.CssSelect(".navbar-brand").First();

This should get you your expected results.

Answer 2

Move your call to a backgroundworker thread. Notice that in line 353 in ScrapingBrowser.cs (ScrapySharp/ScrapySharp/Network/ScrapingBrowser.cs), the call to NavigateToPage() calls the Async version:

public WebPage NavigateToPage(Uri url, HttpVerb verb = HttpVerb.Get, string data = "", string contentType = null)
{
  return NavigateToPageAsync(url, verb, data, contentType).Result;
}

I had the same problem, as soon as I moved the call to my DoWork method in my BGW thread, it starts behaving the way you expect.

Answer 3

Another method would be to use the async version of the NavigateToPage eg:

private async Task<WebPage> LoadPage(Uri uri)
{
    WebPage page = await browser.NavigateToPageAsync(uri);
    return page;
}