How to retry a rake task if you get a Bad Gateway error response from a web source

Question

I am trying to run a rake task to get all the data with a specific tag from Instagram, and then input some of the data into my server.

The task runs just fine, except sometimes I'll get an error response. It's sort of random, so I think it just happens sometimes, and since it's a fairly long running task, it'll happen eventually.

This is the error on my console: Instagram::BadGateway: GET https://api.instagram.com/v1/tags/xxx/media/recent.json?access_token=xxxxx&max_id=996890856542960826: 502: The server returned an invalid or incomplete response.

When this happens, I don't know what else to do except run the task again starting from that max_id. However, it would be nice if I could get the whole thing to automate itself, and retry itself from that point when it gets that error.

My task looks something like this:

task :download => :environment do
  igs = Instagram.tag_recent_media("xxx")
  begin
    sleep 0.2
    igs.each do |ig|
      dl = Instadownload.new
      dl.instagram_url = ig.link
      dl.image_url = ig.images.standard_resolution.url
      dl.caption = ig.caption.text if ig.caption
      dl.taken_at = Time.at(ig.created_time.to_i)
      dl.save!
    end
    if igs.pagination.next_max_id?
      igs = Instagram.tag_recent_media("xxx", max_id: igs.pagination.next_max_id)
      moreigs = true
    else
      moreigs = false
    end
  end while moreigs
end

Answer 1

Chad Pytel and Tammer Saleh call this "Fire and forget" antipattern in their Rails Antipatterns book:

Assuming that the request always succeeds or simply not caring if it fails may be valid in rare circumstances, but in most cases it's unsufficient. On the other hand, rescuing all the exceptions would be a bad practice aswell. The proper solution would be to understand the actual exceptions that will be raised by external service and rescue those only.

So, what you should do is to wrap your code block into begin/rescue block with the appropriate set of errors raised by Instagram (list of those errors can be found here). I'm not sure which particular line of your code snippet ends with 502 code, so just to give you and idea of what it could look like:

begin
  dl = Instadownload.new
  dl.instagram_url = ig.link
  dl.image_url = ig.images.standard_resolution.url
  dl.caption = ig.caption.text if ig.caption
  dl.taken_at = Time.at(ig.created_time.to_i)
  dl.save!
rescue Instagram::BadGateway => e # list of acceptable errors can be expanded
  retry  # restart from beginning
end