Web Crawler issue in perl

Question

I had built a web crawler in Perl.

I am using

HTML::ContentExtractor

LWP::UserAgent 

HTML::LinkExtor

to extract text form webpages.

Reference link for sample code web cralwer perl

Issue:

The issue is that it does not get text from web pages that have the extension as .aspx . It works perfectly for other webpages.I could not figure out the issue why this crawler fails for aspx pages.

Answer 1

To be able to access a process a website with JavaScript, use WWW::Mechanize::Firefox