how to download pdf file with R from web (encode issue)

Question

I am trying to download a pdf file from a website using R. When I tried to to use the function browserURL, it only worked with the argument encodeIfNeeded = T. As a result, if I pass the same url to the function download.file, it returns "cannot open destfile 'downloaded/teste.pdf', reason 'No such file or directory", i.e., it cant find the correct url.

How do I correct the encode, in order for me to be able to download the file programatically? I need to automate this, because there are more than a thousand files to download.

Here's a minimum reproducible code:

library(tidyverse)
library(rvest)


url <- "http://www.ouvidoriageral.sp.gov.br/decisoesLAI.html"
webpage <- read_html(url)

# scrapping hyperlinks
links_decisoes <- html_nodes(webpage,".borderTD a") %>%
  html_attr("href")

# creating full/correct url
full_links <- paste("http://www.ouvidoriageral.sp.gov.br/", links_decisoes, sep="" )

# browseURL only works with encodeIfNeeded = T
browseURL(full_links[1], encodeIfNeeded = T,
          browser = "C://Program Files//Mozilla Firefox//firefox.exe")
# returns an error
download.file(full_links[1], "downloaded/teste.pdf") 

Answer 1

There are a couple of problems here. Firstly, the links to some of the files are not properly formatted as urls - they contain spaces and other special characters. In order to convert them you must use url_escape(), which should be available to you as loading rvest also loads xml2, which contains url_escape().

Secondly, the path you are saving to is relative to your R home directory, but you are not telling R this. You either need the full path like this: "C://Users/Manoel/Documents/downloaded/testes.pdf", or a relative path like this: path.expand("~/downloaded/testes.pdf").

This code should do what you need:

library(tidyverse)
library(rvest)

# scraping hyperlinks
full_links <- "http://www.ouvidoriageral.sp.gov.br/decisoesLAI.html" %>%
               read_html()                                           %>%
               html_nodes(".borderTD a")                             %>%
               html_attr("href")                                     %>%
               url_escape()                                          %>%
               {paste0("http://www.ouvidoriageral.sp.gov.br/", .)}

# Looks at page in firefox
browseURL(full_links[1], encodeIfNeeded = T, browser = "firefox.exe")

# Saves pdf to "downloaded" folder if it exists
download.file(full_links[1], path.expand("~/downloaded/teste.pdf"))