Loop Stops when API returns NULL in R

Question

I am looping through an API to match strings and standardize data as per my own reference data-set. In most cases, the API gives a response and the results are populated in an output file. However, when the API returns a NULL the loop stops and I need to remove the particular string for it to function again. This is a severely iterative process. Is there any way to

1. Find Strings where the API will return NULL? Such strings can be fixed in our data
2. Populate NULL or NA in the output file for strings returning NULL

I cannot share the API as it has been developed internally in the organization but will share the code.

DESTINATIONS<- subset(DESTINATIONS, DESTINATIONS!="ABCDEF")


df <- data.frame()

for(i in 1:nrow(DESTINATIONS))
{
  
  location_url <- paste0(base_url, "destinations?name=", DESTINATIONS(DESTINATIONS))[i],specs)
  
  
  
destination_res <- GET(location_url)
   
destination_text <- content(destination_res, "text", encoding = "UTF-8")
  
location_df1 <- fromJSON(destination_text, flatten = TRUE)
  
location_df1 <- do.call(c, unlist(location_df1, recursive=FALSE))
  
location_df1 <- as.data.frame(t(location_df1))

coordinates_a <- select(location_df1, contains("items.country.name"))
  
coordinates_a <- coordinates_a %>% distinct() %>% t()
  
coordinates_a <- as.data.frame(coordinates_a)
  
coordinates_b <- select(location_df1, contains("items.id"))
  
coordinates_b <- coordinates_b %>% distinct() %>% t()
  
coordinates_b <- as.data.frame(coordinates_b)

coordinates <-  cbind.data.frame(coordinates_a, coordinates_b)

df <- rbind.data.frame(df, coordinates)}

In short, if a string from DESTINATIONS dataframe does not have a response from the API, the loop breaks

Thank you for the all the help in advance.

Answer 1

Assuming the GET request would return NULL you can check for it's length.

Try using something like this :

data_list <- vector('list', nrow(DESTINATIONS))

for(i in 1:nrow(DESTINATIONS)) {
   location_url <- paste0(base_url, "destinations?name=", 
                          DESTINATIONS(DESTINATIONS))[i],specs)
   destination_res <- GET(location_url)
   if(length(destination_res) > 0) {
      destination_text <- content(destination_res, "text", encoding = "UTF-8")
      location_df1 <- fromJSON(destination_text, flatten = TRUE)
      location_df1 <- do.call(c, unlist(location_df1, recursive=FALSE))
      location_df1 <- as.data.frame(t(location_df1))
      coordinates_a <- select(location_df1, contains("items.country.name"))
      coordinates_a <- coordinates_a %>% distinct() %>% t()
      coordinates_a <- as.data.frame(coordinates_a)
      coordinates_b <- select(location_df1, contains("items.id"))
      coordinates_b <- coordinates_b %>% distinct() %>% t()
      coordinates_b <- as.data.frame(coordinates_b)
      coordinates <-  cbind.data.frame(coordinates_a, coordinates_b)
     }
   else coordinates <- NA
   data_lst[[i]] <- coordinates
}

Now, the requests which failed can be found by :

which(is.na(data_lst))

and to bind the data together you can remove those NA values.

complete_data <- do.call(rbind, Filter(is.data.frame, data_lst))