Warming Up Cache Digests Overnight

Question

We have a Rails 3.2 website which is fairly large with thousands of URLs. We implemented Cache_Digests gem for Russian Doll caching. It is working well. We want to further optimize by warming up the cache overnight so that user gets a better experience during the day. I have seen answer to this question: Rails: Scheduled task to warm up the cache?

Could it be modified for warming up large number of URLs?

Answer 1

To trigger cache hits for many pages with expensive load times, just create a rake task to iteratively send web requests to all record/url combinations within your site. (Here is one implementation)

Iteratively Net::HTTP request all site URL/records:

To only visit every page, you can run a nightly Rake task to make sure that early morning users still have a snappy page with refreshed content.

lib/tasks/visit_every_page.rake:

namespace :visit_every_page do
  include Net
  include Rails.application.routes.url_helpers

  task :specializations => :environment do
    puts "Visiting specializations..."
    Specialization.all.sort{ |a,b| a.id <=> b.id }.each do |s|
      begin
        puts "Specialization #{s.id}"
        
        City.all.sort{ |a,b| a.id <=> b.id }.each do |c|
          puts "Specialization City #{c.id}"
          Net::HTTP.get( URI("http://#{APP_CONFIG[:domain]}/specialties/#{s.id}/#{s.token}/refresh_city_cache/#{c.id}.js") )
        end
      
        Division.all.sort{ |a,b| a.id <=> b.id }.each do |d|
          puts "Specialization Division #{d.id}"
          Net::HTTP.get( URI("http://#{APP_CONFIG[:domain]}/specialties/#{s.id}/#{s.token}/refresh_division_cache/#{d.id}.js") )
        end
      end
    end
  end

  # The following methods are defined to fake out the ActionController
  # requirements of the Rails cache
  
  def cache_store
    ActionController::Base.cache_store
  end

  def self.benchmark( *params )
    yield
  end

  def cache_configured?
    true
  end
end

(If you want to directly include cache expiration/recaching into this task, check out this implementation.)

via a Custom Controller Action:

If you need to bypass user authentication restrictions to get to your pages, and/or you don't want to screw up (too badly) your website's tracking analytics, you can create a custom controller action for hitting cache digests that use tokens to bypass authentication:

app/controllers/specializations.rb:

class SpecializationsController < ApplicationController
...
  before_filter :check_token, :only => [:refresh_cache, :refresh_city_cache, :refresh_division_cache]
  skip_authorization_check :only => [:refresh_cache, :refresh_city_cache, :refresh_division_cache]

...

  def refresh_cache
    @specialization = Specialization.find(params[:id])
    @feedback = FeedbackItem.new
    render :show, :layout => 'ajax'
  end

  def refresh_city_cache
    @specialization = Specialization.find(params[:id])
    @city = City.find(params[:city_id])
    render 'refresh_city.js'
  end

  def refresh_division_cache
    @specialization = Specialization.find(params[:id])
    @division = Division.find(params[:division_id])
    render 'refresh_division.js'
  end

end

Our custom controller action renders the views of other expensive to load pages, causing cache hits to those pages. E.g. refresh_cache renders the same view page & data as controller#show, so requests to refresh_cache will warm up the same cache digests as controller#show for those records.

Security Note:

For security reasons, I recommend before providing access to any custom refresh_cache controller request that you pass in a token and check it to make sure that it corresponds with a unique token for that record. Matching URL tokens to database records before providing access (as seen above) is trivial because your Rake task has access to the unique tokens of each record -- just pass the record's token in with each request.

tl;dr:

To trigger thousands of site URL's/cache digests, create a rake task to iteratively request every record/url combination in your site. You can bypass your app's user authentication restrictions for this task by creating a a custom controller action that authenticates access via tokens instead.

Answer 2

I realize this question is about a year old, but I just worked out my own answer, after scouring a bunch of partial & incorrect solutions.

Hopefully this will help the next person...

Per my own utility class, which can be found here: https://raw.githubusercontent.com/JayTeeSF/cmd_notes/master/automated_action_runner.rb

You can simply run this (per it's .help method) and pre-cache your pages, without tying-up your own web-server, in the process.

class AutomatedActionRunner  
  class StatusObject
    def initialize(is_valid, error_obj)
      @is_valid = !! is_valid
      @error_obj = error_obj
    end

    def valid?
      @is_valid
    end

    def error
      @error_obj
    end
  end

  def self.help
    puts <<-EOH
      Instead tying-up the frontend of your production site with:
        `curl http://your_production_site.com/some_controller/some_action/1234`
        `curl http://your_production_site.com/some_controller/some_action/4567`
      Try:
        `rails r 'AutomatedActionRunner.run(SomeController, "some_action", [{id: "1234"}, {id: "4567"}])'`
    EOH
  end

  def self.common_env
    {"rack.input"  => "", "SCRIPT_NAME" => "", "HTTP_HOST" => "localhost:3000" }
  end
  REQUEST_ENV = common_env.freeze

  def self.run(controller, controller_action, params_ary=[], user_obj=nil)
    success_objects = []
    error_objects = []
    autorunner = new(controller, controller_action, user_obj)
    Rails.logger.warn %Q|[AutomatedAction Kickoff]: Preheating cache for #{params_ary.size} #{autorunner.controller.name}##{controller_action} pages.|

    params_ary.each do |params_hash|
      status = autorunner.run(params_hash)
      if status.valid?
        success_objects << params_hash
      else
        error_objects << status.error
      end
    end

    return process_results(success_objects, error_objects, user_obj.try(:id), autorunner.controller.name, controller_action)
  end

  def self.process_results(success_objects=[], error_objects=[], user_id, controller_name, controller_action)
    message = %Q|AutomatedAction Summary|
    backtrace = (error_objects.first.try(:backtrace)||[]).join("\n\t").inspect
    num_errors = error_objects.size
    num_successes = success_objects.size

    log_message = %Q|[#{message}]: Generated #{num_successes} #{controller_name}##{controller_action}, pages; Failed #{num_errors} times; 1st Fail: #{backtrace}|
    Rails.logger.warn log_message

    # all the local-variables above, are because I typically call Sentry or something with extra parameters!
  end

  attr_reader :controller
  def initialize(controller, controller_action, user_obj)
    @controller = controller
    @controller = controller.constantize unless controller.respond_to?(:name)
    @controller_instance = @controller.new
    @controller_action = controller_action
    @env_obj = REQUEST_ENV.dup
    @user_obj = user_obj
  end

  def run(params_hash)
    Rails.logger.warn %Q|[AutomatedAction]: #{@controller.name}##{@controller_action}(#{params_hash.inspect})|
    extend_with_autorun unless @controller_instance.respond_to?(:autorun)

    @controller_instance.autorun(@controller_action, params_hash, @env_obj, @user_obj)
  end


  private

  def extend_with_autorun
    def @controller_instance.autorun(action_name, action_params, action_env, current_user_value=nil)
      self.params = action_params # suppress strong parameters exception
      self.request = ActionDispatch::Request.new(action_env)
      self.response = ActionDispatch::Response.new
      define_singleton_method(:current_user, -> { current_user_value })

      send(action_name) # do it
      return StatusObject.new(true, nil)
    rescue Exception => e
      return StatusObject.new(false, e)
    end
  end
end