I have a RoR application running on two different server. They run the same version of the app and are similar in configuration. I have a Mongodb replica set running on both the server with a third server as an arbitrary server.
Everything runs fine. The data is syncing perfectly. But after 2 weeks of running, one of the server started giving ConnectionPool::PoolShuttingDownError. I checked the log and I can see the error was raised application controller. I didn't change any code on any of the server.
The server raising the error is fine till it gets 6-7 simultaneous request. Or when you refresh the page 6-7 times together. It gives this error once and then again you refresh the page and its back to normal. This is weird and I can't understand why one server has this problem while the other one don't and that too sometime.
I am using Mongoid with Moped, Rails 4.1.0 and Ruby 2.1.5. I also checked available connections using db.serverStatus().connections which is around 51158 and ulimit of max process is 257185.
I searched a lot but I am still unsure of the cause of this problem. It will be great if someone can put some light on this issue. Any help will be appreciated. Thanks in advance.