Can I use Perl LWP::UserAgent to scrape HTML and rendered JavaScript source?

1.3k Views Asked by At

Is it possible to use Perl LWP::UserAgent to scrape HTML and rendered JavaScript source?

1

There are 1 best solutions below

2
On

LWP::UserAgent will let you simulate a browser within Perl, so it'll fetch things via HTTP for you (html, javascript, images, various binary files, etc...). But it won't execute the Javascript for you. That's not its job.

In real world terms, LWP::UA is a delivery truck. The fact that it's hauling kitchen equipment won't let LWP::UA cook dinner for you. It just drops the stove off at your front door.