Malformed JSON string when using perl and earthexplorer

Question

I'm currently working with the USGS's EarthExplorer to setup some batch downloads of Landsat scenes based on spatial coordinates. They provide a wonderful handy script to do this at https://earthexplorer.usgs.gov/inventory/example/json-download_data-pl which is great. I'm working on a cluster, and despite installing all perl modules properly, when I run the script I get the following output:

Running Script...


Error: Error: malformed JSON string, neither array, object, number, string or 
atom, at character offset 0 (before "LWP will support htt...") at ./dl.pl line 182

This seems curious. As way of explanation, the script starts out with

#!/usr/bin/perl

#use strict;
use warnings;
use 5.010;
use JSON;
use Scalar::Util qw(looks_like_number reftype dualvar );
use LWP::UserAgent;
use Getopt::Long qw(GetOptions);

my ($username, $password);
$username = "myusername_filled_in";
$password = "mypassword_filled_in";

GetOptions(
    'username=s' => \$username,
    'password=s' => \$password,
) or die "Error retrieving Username and Password\n";

and the offending bit of code is

    $res = $response->{_content};
    $res = decode_json $res;

Following very useful advice in Can't run Perl script on other computer I have done the following:

1. Changed $response->content to $response->decoded_content( charset => 'none') in the offending area of code.

2. Ran lwp-request https://google.com/ which just pulled back a full webpage - no error. So, that seems to be working.

3. Tried to see some debug by inserting print $response->decoded_content( charset => 'none'); which then threw the error

LWP will support https URLs if the LWP::Protocol::https module is installed.

And, indeed, LWP::Protocol::https is installed.

I feel like there must be something simple I'm missing - something like how I defined my username and password (just $username = "myusername"; etc., after the variables are declared) or something else asinine.

Has anyone else run into this?

---

To add output from the query below:

$ which cpan ; head -n 1 `which cpan` ; echo 'o conf' | cpan | grep -P 'make|mbuild' ; set | grep ^PERL ; which perl ; perl -le'use LWP::Protocol::https; print "ok";'

/share/pkg/perl/5.10.1/bin/cpan
#!/share/pkg/perl/5.10.1/bin/perl
    make               [/usr/bin/make]
    make_arg           []
    make_install_arg   []
    make_install_make_command [/usr/bin/make]
    makepl_arg         []
    mbuild_arg         []
    mbuild_install_arg []
    mbuild_install_build_command [./Build]
    mbuildpl_arg       []
PERL5LIB=/home/jb92b/perl5/lib/perl5:/home/jb92b/perl5/lib/perl5:/home/jb92b/perl5/lib/perl5
PERL_LOCAL_LIB_ROOT=/home/jb92b/perl5:/home/jb92b/perl5:/home/jb92b/perl5
PERL_MB_OPT='--install_base "/home/jb92b/perl5"'
PERL_MM_OPT=INSTALL_BASE=/home/jb92b/perl5
/share/pkg/perl/5.10.1/bin/perl
ok

Answer 1

Your code doesn't check for errors. You should have something like

$response->is_success()
   or die("Can't fetch X: ".$response->status_line());

The problem you are encountering is that LWP::Protocol::https isn't installed.

You claim it is, but Perl is authoritative here :) It wasn't installed by that Perl, you installed it a non-standard directory without tell Perl to look for it there, or there's a permission issue.

In this case, your script uses /usr/bin/perl, but you installed the module using/for /share/pkg/perl/5.10.1/bin/perl. You need to switch which perl your script uses, or you need to install the module using/for /usr/bin/perl.

Answer 2

"something like how I defined my username and password (just $username = "myusername"; etc., after the variables are declared)"

You shouldn't modify the code to add your username and password. The way it's written, the call to GetOptions will overwrite them. You should use

program.pl --usernam myusername --password mypassword