speed up response of request with cache or binary response

Question

I want to speed up response of query with cache or binary response

It takes more than 30 seconds ~ 500 seconds to render a json response for 5k ~ 100k records.

How to make it more quickly ?

Because the response time is still slow on rendering

Is it possible to make a binary response (skip render view ) ?

  WeatherLog Load (335.5ms)  SELECT  "weather_logs".* FROM "weather_logs"   ORDER BY "weather_logs"."datetime" ASC LIMIT 90000
Write page /Users/public/index.json (35.2ms)
Completed 200 OK in 40488ms (Views: 32459.3ms | ActiveRecord: 336.8ms)

Controller

caches_page :index
respond_to :json

def index
  begin       
    @weather_logs = WeatherLog.result(q)

    respond_to do |format|
      render json: @weather_logs
      return
    end
  rescue Exception => e
    respond_to do |format|
      format.json { render json: {status: "Invalid Request #{e.backtrace.first(3)}"} }
    end
  end
end

Dataflow : Step 4 is the bottleneck for rendering

In short words

I only need to send the response to client in a short time,

No matter what format it is.

Even binary format is ok.

Please give me some direction or idea to boost the performance

Answer 1

Is it definitely just the rendering that is the problem, what is the result of WeatherLogs.result(q). Have you benchmarked the WeatherLogs#results call? How long does it take to iterate over that collection returned opposed to just rendering it? Are there any methods being called when rendering it to JSON e.g. not only class fields?

If it's definitely the rendering that is the problem you might want to consider using a LRU cache when inserting the data into WeatherLogs, essentially denormalizing the data that is being requested by the clients. Because, depending on your traffic and the amount of queries possible(although this will also influence the denormalized data) it will almost never hit the cache when it's generated on render time. This article might be a good start: http://oldblog.antirez.com/post/redis-as-LRU-cache.html

But without understanding exactly why the rendering takes so long I wouldn't go as far as building a sophisticated cache, if JSON rendering is the problem OJ might be a solution or whatever lib is claiming to be the fastest but I have a hunch it's not the actual rendering that's the problem.

Happy o help if you have any further questions.

Answer 2

You need to call Oj explicitly:

  def index
    begin         
      @weather_logs = WeatherLog.result(q)

      respond_to do |format|
        render json: Oj.dump(@weather_logs)
        return
      end

And if you want to make it implicit, try this gem by my company: https://github.com/GoodLife/rails-patch-json-encode