Extracting all links of a certain form

Question

I've got a page that I want all the links off of (e.g. http://www.stephenfry.com/). I want to put all the links that are of the form http://www.stephenfry.com/WHATEVER into an array. What I've got now is just the following method:

#!/usr/bin/perl -w
use strict;
use LWP::Simple;
use HTML::Tree;

# I ONLY WANT TO USE JUST THESE

my $url = 'http://www.stephenfry.com/';

my $doc = get( $url );

my $adt = HTML::Tree->new();
$adt->parse( $doc );

my @juice = $adt->look_down(
    _tag => 'a',
    href => 'REGEX?'
);

Not sure how to put only these links in.

Answer 1

You'll want to use the extract_links() method, not look_down():

use strict;
use warnings;
use LWP::Simple;
use HTML::Tree;

my %seen;
my $url = 'http://www.stephenfry.com/';
my $doc = get($url);

my $adt = HTML::Tree->new();
$adt->parse($doc);
my $links_array_ref = $adt->extract_links('a');

my @links = grep { /www.stephenfry.com/ and !$seen{$_}++ } map $_->[0],
  @$links_array_ref;

print "$_\n" for @links;

Partial output:

http://www.stephenfry.com/
http://www.stephenfry.com/blog/
http://www.stephenfry.com/category/blessays/
http://www.stephenfry.com/category/features/
http://www.stephenfry.com/category/general/
...

Using WWW::Mechanize may be simpler, and it does return more links:

use strict;
use warnings;
use WWW::Mechanize;

my %seen;
my $mech = WWW::Mechanize->new();
$mech->get('http://www.stephenfry.com/');
my @links = grep { /www.stephenfry.com/ and !$seen{$_}++ } map $_->url,
  $mech->links();

print $_, "\n" for @links;

Partial output:

http://www.stephenfry.com/wp-content/themes/fry/images/favicon.png
http://www.stephenfry.com/wp-content/themes/fry/style.css
http://www.stephenfry.com/wordpress/xmlrpc.php
http://www.stephenfry.com/feed/
http://www.stephenfry.com/comments/feed/
...

Hope this helps!