rails too many open files using Feedzirra

Question

I am getting a LoadError - "Too many open files" when using Feedzirra. I am running it on my development server using the default WEBrick server.

I am parsing only 2 feeds. What is the problem?

Answer 1

I had the same issue with Feedzirra. You can notice that it leaves TCP connections in CLOSE_WAIT state forever, hence causing the problem.

It appears to be curb gem specific that is used to fetch feeds. Another project depending on libcurl had the same issue. They have fixed it by setting 'CURLOPT_FORBID_REUSE' option.

I've tried to do the same for Feedzirra but didn't succeed. Even with this option I had a growing number of CLOSE_WAIT sessions and Too many open files error eventually.

So I did the most straightforward thing, I download feeds using Net::HTTP:

def get_contents(furl)
  url = URI.parse(furl)
  req = Net::HTTP::Get.new(url.to_s)

  res = Net::HTTP.start(url.host, url.port) { |http|
    http.request(req)
  }

  unless res.kind_of? Net::HTTPSuccess
    puts "can't get feed #{url.to_s}: #{res.code}"
    return nil
  end

  res.body
end

Then I parse the XML with Feedzirra:

xml = get_contents(furl)
feedin = Feedzirra::Feed.parse xml

No more stuck connections and no more errors. You may also want to add better error handling to this sample code.