Randomizing job scheduling with cron and sidekiq

Question

I am building a small crawler/scraper for my dad! I am trying to randomize scrape times every week (meaning I want the scrape method to fire off every week at a different day and time of day).

I tried the rewrite the YAML file every time the method fires off that does not seem to work... Then I tried the "set" method:

def set_random_schedule(author_name)
  cron_expression = random_cron_expression
  self.class.set(cron: cron_expression).perform_async(author_name)
end

def random_cron_expression
  "#{rand(1..3)} * * * *}"
end

In the logs it said it was running almost every second, I honestly don't know why... Or if I read it wrong I am really new to sidekiq and cron so I am not sure where did I go wrong. Bare with me here please.

Any help to beginner? Feel free to explain or reference whatever comes to mind I am always keen to learn.

Answer 1

At the end of your job, you can try to add this code to reschedule its cron timer:

scrape_job = Sidekiq::Cron::Job.all.find{ |job| job.klass == "YourClassNameJob"}
scrape_job.cron = "#{rand(1..3)} * * * *}"
scrape_job.save

Answer 2

If I were you I wouldn't be using cron at all. Just have some code at the end of your perform that schedules another sidekiq task for the future at your randomized time. Like:

def perform(your, args)

  ## do your stuff

  ThisModule.perform_at(random_time, your, args)
end

def random_time
  next_week = 1.week.from_now
  start_of_nw = next_week.beginning_of_week
  seconds_in_nw = ( start_of_nw + 1.week ) - start_of_nw

  start_of_nw + rand(seconds_in_nw)
end

Note that the random_time is a little more complex than you might expect in order to account for daylight savings switches.