Environment
I have a rails 3.2.x app load balanced between 2 webservers: web1 and web2. Each webserver points to a clustered IP for the mysql database. The database is 2 servers clustered (by rackspace). 1 database is active at a time until failover.
Error
On our latest failover web1 started sending ActiveRecord::ConnectionTimeoutError on Middleware/Rack/ActiveRecord::QueryCache#call (stack trace below) while web2 had no errors. Note, this occurred after the DB was stable and 100% switched to the other node.
Theory
My theory is that the session or cache or connection referenced the old node and once switched it could not find said connection. This does seem odd since both DB nodes are clustered, serving on the same clustered IP and are replicated.
Temporary solution
Our interim solution was to restart web1 rails server, this worked.
Questions
1) I was wondering if I could get an explanation of what might be happening here
2) Is there a configuration setting that could prevent this in the future.
Stack Trace
Error message ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5 seconds (waited 5.000268907 seconds). The max pool size is currently 5; consider increasing it.
…record/connection_adapters/abstract/connection_pool.rb: 258:in block (2 levels) in checkout'
…record/connection_adapters/abstract/connection_pool.rb: 242:inloop'
…record/connection_adapters/abstract/connection_pool.rb: 242:in block in checkout'
/usr/local/lib/ruby/1.9.1/monitor.rb: 211:inmon_synchronize'
…record/connection_adapters/abstract/connection_pool.rb: 239:in checkout'
…record/connection_adapters/abstract/connection_pool.rb: 102:inblock in connection'
/usr/local/lib/ruby/1.9.1/monitor.rb: 211:in mon_synchronize'
…record/connection_adapters/abstract/connection_pool.rb: 101:inconnection'
…record/connection_adapters/abstract/connection_pool.rb: 410:in retrieve_connection'
…nnection_adapters/abstract/connection_specification.rb: 171:inretrieve_connection'
…nnection_adapters/abstract/connection_specification.rb: 145:in connection'
…s/activerecord-3.2.19/lib/active_record/query_cache.rb: 67:inrescue in call'
…s/activerecord-3.2.19/lib/active_record/query_cache.rb: 61:in call'
…record/connection_adapters/abstract/connection_pool.rb: 479:incall'
…ack-3.2.19/lib/action_dispatch/middleware/callbacks.rb: 28:in block in call'
…s/activesupport-3.2.19/lib/active_support/callbacks.rb: 405:in_run__862588711471957735__call__1324336831638990183__callbacks'
…s/activesupport-3.2.19/lib/active_support/callbacks.rb: 405:in __run_callback'
…s/activesupport-3.2.19/lib/active_support/callbacks.rb: 385:in_run_call_callbacks'
…s/activesupport-3.2.19/lib/active_support/callbacks.rb: 81:in run_callbacks'
…ack-3.2.19/lib/action_dispatch/middleware/callbacks.rb: 27:incall'
…ack-3.2.19/lib/action_dispatch/middleware/remote_ip.rb: 31:in call'
…/gems/airbrake-3.1.14/lib/airbrake/rails/middleware.rb: 13:incall'
….19/lib/action_dispatch/middleware/debug_exceptions.rb: 16:in call'
…2.19/lib/action_dispatch/middleware/show_exceptions.rb: 56:incall'
…by/1.9.1/gems/railties-3.2.19/lib/rails/rack/logger.rb: 32:in call_app'
…by/1.9.1/gems/railties-3.2.19/lib/rails/rack/logger.rb: 16:inblock in call'
…ivesupport-3.2.19/lib/active_support/tagged_logging.rb: 22:in tagged'
…by/1.9.1/gems/railties-3.2.19/lib/rails/rack/logger.rb: 16:incall'
…ck-3.2.19/lib/action_dispatch/middleware/request_id.rb: 22:in call'
…/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb: 21:incall'
…/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb: 17:in call'
…9.1/gems/airbrake-3.1.14/lib/airbrake/user_informer.rb: 16:in_call'
…9.1/gems/airbrake-3.1.14/lib/airbrake/user_informer.rb: 12:in call'
…le/ruby/1.9.1/gems/railties-3.2.19/lib/rails/engine.rb: 484:incall'
…by/1.9.1/gems/railties-3.2.19/lib/rails/application.rb: 231:in call'
…gems/railties-3.2.19/lib/rails/railtie/configurable.rb: 30:inmethod_missing'
…r-3.0.13/lib/phusion_passenger/rack/request_handler.rb: 96:in `process_request'