Extracting multiple keywords (in loop) for multiple countries in google trends

Question

I am using the gtrendsR package for extracting google trends data. I understand that this package has a limit of maximum 5 "keywords" at a time; therefore I used a loop to extract >5 "keywords" at a time.

Now i would like to repeat this excercise for multiple "countries" as well, and would like the result to show all possible google trend outputs for every combination of the "keyword" and "country"

This is the code i am using:

Country = readLines("country_list.csv")
Keyword = readLines("keyword_list.csv")
results <- list() 

for (i in Keyword) 
  {
  for (j in Country) {
    time=("today 3-m")
    channel='web'
    trends = gtrends(keyword=i, gprop =channel,geo=j, time = time)
    results [[j]] <- trends$interest_over_time
  }
} 

Out <- as.data.frame(do.call("rbind", results)) 

I keep getting the error:

Error in get_widget(comparison_item, category, gprop, hl, cookie_url, : widget$status_code == 200 is not TRUE

I have around 60 "countries" and 300 "keywords" on the lists. Is it down to excess data extraction not being possible from Google Trends ? Or some fundamental error

Btw, i am a basic user of R; so many thanks for the help

Answer 1

The error codes, widget$status_code == 200, the server returns are usually pretty descriptive of the problem if you just put them into a google search. In your case, too many requests in a short period of time. For every i in keyword you are calling to the server length(Country) times. that's a lot of requests in a short period & you're going to get blocked. Either set a some type of time out in-between your calls or look into hacky scraper methods like rotating headers/cookies etc.

Answer 2

I solved by downloading gtrendsR from:

install.packages("remotes")
remotes::install_github("PMassicotte/gtrendsR")

(see https://github.com/PMassicotte/gtrendsR/issues/166)

and using the following

Country = readLines("states.csv")
Keyword = readLines("celebs.csv")


for (i in Keyword) 
{
  for (j in Country) {
    time=("2018-01-01 2018-06-30")
    channel='web'
    trends = gtrends(keyword=i, gprop =channel,geo=j, time = time, onlyInterest = TRUE,low_search_volume = FALSE)
    Sys.sleep(5)
    results [[j]][[i]] <- trends$interest_over_time
  }
}