Get data from OpenDap server that requires authentication using R

Question

I'm trying to get data from an OPeNDAP server using R and the ncdf4 package. However, the nasa eosdis server requires username / password. How can I pass this info using R?

Here is what I'm trying to do:

require(ncdf4)
f1 <- nc_open('https://disc2.gesdisc.eosdis.nasa.gov/opendap/TRMM_L3/TRMM_3B42.7/2018/020/3B42.20180120.15.7.HDF')

And the error message:

Error in Rsx_nc4_get_vara_double: NetCDF: Authorization failure syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: HTTP^ Basic: Access denied. Var: nlat Ndims: 1
Start: 0 Count: 400 Error in ncvar_get_inner(d$dimvarid$group_id, d$dimvarid$id, default_missval_ncdf4(), : C function R_nc4_get_vara_double returned error

I tried the url https://username:password@disc2.... but that did not work also.

Answer 1

Daniel,

The service you are accessing is using third-party redirection to authenticate users. Therefore the simple way of providing credentials in the URL doesn't work.

You need to create 2 files.

1. A .dodsrc file (a RC file for the netcdf-c library) with the following content

   HTTP.COOKIEFILE=.cookies
   HTTP.NETRC=.netrc

2. A .netrc file, in the location referenced in the .dodsrc, with your credentials:

   machine urs.earthdata.nasa.gov
   login YOURUSERNAMEHERE
   password YOURPASWORDHERE

You can find more details at https://www.unidata.ucar.edu/software/netcdf/docs/md__Users_wfisher_Desktop_v4_86_81-prep_netcdf-c_docs_auth.html

Regards

Antonio

Answer 2

unfortunately, even after defining the credentials and their location

ncdf4::nc_open("https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGDE.06/2020/08/3B-DAY-E.MS.MRG.3IMERG.20200814-S000000-E235959.V06.nc4")

still returns

Error in Rsx_nc4_get_vara_double: NetCDF: Authorization failure

The same happens when using ncdump from a terminal:

$ ncdump https://gpm1.gesdisc.eosdis.nasa.gov/opendap/GPM_L3/GPM_3IMERGDE.06/2020/08/3B-DAY-E.MS.MRG.3IMERG.20200814-S000000-E235959.V06.nc4

returns

syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: HTTP^ Basic: Access denied. NetCDF: Authorization failure Location: file /build/netcdf-KQb2aQ/netcdf-4.6.0/ncdump/vardata.c; line 473