modrails - rogue ruby processes consuming 100% cpu

Question

I'm having ruby instances from mod_rails go "rogue" -- these processes are no longer listed in passenger-status and utilize 100% cpu.

Other than installing god/monit to kill the instance, can anyone give me some advice on how to prevent this? I haven't been able to find anything in the logs that helps.

Answer 1

I had a ruby process related to Phusion Passenger, which consumed lots of CPU, even though it should have been idle.

The problem went away after I ran

date -s "`date`"

as suggested in this thread. (That was on Debian Squeeze)

Apparently, the problem was related to a leap second, and could affect many other applications like MySQL, Java, etc. More info in this thread on lklm.

Answer 2

This is a recurring issue with passenger. I've seen this problem many times helping people that ran ruby on rails with passenger. I don't have a fix but you might want to try this http://www.modrails.com/documentation/Users%20guide%20Apache.html#debugging_frozen

Answer 3

We saw something similar to this with very long running SQL queries.

MySQL would kill the queries because they exceeded the long running limit and the thread never realized that the query was dead.

You may want to check the database logs.

Answer 4

If you're using Linux, you can install the "strace" utility to see what the Ruby process is doing that's consuming all the CPU. That will give you a good low-level view. It should be available in your package manager. Then you can:

$ sudo strace -p 22710
Process 22710 attached - interrupt to quit
...lots of stuff...
(press Ctrl+C)

Then, if you want to stop the process in the middle and dump a stack trace, you can follow the guide on using GDB in Ruby at http://eigenclass.org/hiki.rb?ruby+live+process+introspection, specifically doing:

gdb --pid=(ruby process)
session-ruby
stdout_redirect
(in other terminal) tail -f /tmp/ruby_debug.(pid)
eval "caller"

You can also use the ruby-debug Gem to remotely connect to debug sockets you open up, described in http://duckpunching.com/passenger-mod_rails-for-development-now-with-debugger

There also seems to be a project on Github concerned with debugging Passenger instances that looks interesting, but the documentation is lacking: http://github.com/ddollar/socket-debugger/tree/master