How to extract only URLs from Dmoz ODP file (in RDF)

Question

I need only the URL's from the dmoz/ODP file. But the file is in RDF. How do I get only the url's from the odp file? I want to extract all the url's in a text file.

Anyone knows of any script to parse only urls from rdf file ?

Answer 1

Maybe something like this then?

#!/usr/bin/perl
use strict;
use warnings;

my $file = "kt-content.rdf.u8";
my @urls;

open(my $fh, "<", $file) or die "Unable to open $file\n";

while (my $line = <$fh>) {
    if ($line =~ m/<(?:ExternalPage about|link r:resource)="([^\"]+)"\/?>/) {
        push @urls, $1;
    }
}

close $fh;

And then print the contents of @urls to a text file.

Answer 2

Several of the popular SemWeb APIs (Jena, Sesame and dotNetRDF) all provide fully streaming APIs for RDF files so you can write a custom data handler that will only take the URIs produced and throw away the rest of the stuff you aren't interested in.

You can probably do something hacky with perl and it may be faster but it may not be entirely accurate particularly if the RDF uses relative URIs which need to be resolved

Answer 3

Option 1. Download dmoz_v3.zip from http://sourceforge.net/projects/dmoz2mysql/files/latest/download. This is a PHP script that is used to parse the DMOZ RDF data dump files automatically. It features downloading of the files, extracting, cleaning, parsing and inserting the data into a MySQL database.

Option 2. Use the following link to find the tools to extract URLs from RDF dumpfiles

http://www.dmoz.org/Computers/Internet/Searching/Directories/Open_Directory_Project/Use_of_ODP_Data/Upload_Tools/