How to Use RCurl or RMongo via HTTP with Authentication and Self Signed SSL to Read in JSON Data

Question

I am using R to write a program and perform some analyses. The data is being captured by an outside vendor with MongoDB in JSON format. They are providing it to me via a URI on port 443, which they want me to query using cURL. They have authentication in place and self signed SSL.

I can authenticate and dump the data via curl in Windows, however to create a long term sustainable solution it needs to all be done within R.

The vendor says that RCurl "should" work but they aren't providing any support and they basically just don't like the idea of using RMongo and have no comment on it (but if we could make it work that would be awesome, in my opinion).

I have the following packages loaded - ggplot2 - DBI - rjson - RJSONIO (I sometimes don't load this one if I'm using rjson, or visa versa) - RMongo - rstudio - RCurl

The self signed certificate caused issues even with curl, but those were resolved by editing settings in Ruby and then launching a cmd shell with Ruby and using curl that way. I'm not sure if the problems in R are related.

When trying to go the RCurl route I end up with commands/errors like this:

  x <- getURL("https://xxx.xx.xxx.xxx:443/db/_authenticate", userpwd="xxxx:xxxxx") }{Error in function (type, msg, asError = TRUE)  : couldn't connect to host

and when trying to use RMongo I'm even more clueless...

> mongo <- mongoDbConnect("xxx.xx.xxx.xxx")

username = "xxxx" password="xxxxxxxxxxxxx" authenticated <- dbAuthenticate(mongo, username, password) Feb 25, 2013 4:00:09 PM com.mongodb.DBTCPConnector fetchMaxBsonObjectSize WARNING: Exception determining maxBSON size using0 java.io.IOException: couldn't connect to [/127.0.0.1:27017] bc:java.net.ConnectException: Connection refused: connect at com.mongodb.DBPort.open(DBPort.java:224) at com.mongodb.DBPort.go(DBPort.java:101) at com.mongodb.DBPort.go(DBPort.java:82) at com.mongodb.DBPort.findOne(DBPort.java:142) at com.mongodb.DBPort.runCommand(DBPort.java:151) at com.mongodb.DBTCPConnector.fetchMaxBsonObjectSize(DBTCPConnector.java:429) at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:416) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:193) at com.mongodb.DBApiLayer$MyCollection._find(DBApiLayer.java:303) at com.mongodb.DB.command(DB.java:159) at com.mongodb.DB.command(DB.java:144) at com.mongodb.DB._doauth(DB.java:503) at com.mongodb.DB.authenticate(DB.java:440) at rmongo.RMongo.dbAuthenticate(RMongo.scala:24)

Error in .jcall(rmongo.object@javaMongo, "Z", "dbAuthenticate", username,  : 
com.mongodb.MongoException$Network: can't call something
Feb 25, 2013 4:00:10 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to 127.0.0.1:27017 b/c of error
java.io.IOException: couldn't connect to [/127.0.0.1:27017]     bc:java.net.ConnectException: Connection refused: connect
at com.mongodb.DBPort._open(DBPort.java:224)
at com.mongodb.DBPort.go(DBPort.java:101)
at com.mongodb.DBPort.go(DBPort.java:82)
at com.mongodb.DBPort.call(DBPort.java:72)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:202)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DB.command(DB.java:159)
at com.mongodb.DB.command(DB.java:144)
at com.mongodb.DB._doauth(DB.java:503)
at com.mongodb.DB.authenticate(DB.java:440)
at rmongo.RMongo.dbAuthenticate(RMongo.scala:24)

any help would be greatly appreciated!

Answer 1

I had an issue in the past with RCurl where I needed to explicitly point it toward the security certificates to get it to work okay. I ended up needing something like this:

out <- postForm("https://url.org/api/",
                 token="IMATOKEN",
                 .opts=curlOptions(cainfo="C:/path/aaa.crt"))

I had manually exported the certificate I needed to get that working.

Also, it kind of looks like you should be doing a POST request given that URI, not a GET. Try the postForm() command, maybe?

EDITED TO ADD:

Okay, I think things might be a little more clear if we stepped back a second. Is your goal to get some file from a specific URL (basically, doing a wget but from within R)? Or is your goal to submit a form that subsequently returns the data you need?

IF you are just trying to get something that is behind basic (and also fairly INSECURE) HTTP authentication, you should do two things:

• Tell your data provider to use a more secure option
• Use the getURL() option as shown (using the www.omegahat.org example you posted about):

Code:

getURL("http://www.omegahat.org/RCurl/testPassword/",.opts=list(userpwd="bob:welcome"))
OR
getURL("http://bob:[email protected]/RCurl/testPassword/")

Now, if you need to submit a form to get the data, you would generally pass authentication tokens, etc, as parameters (so, in the example above, `token='.