SocketException in Mongo

Question

I just set up a replica set in Mongo (prod environment). I'm now getting a lot of exceptions like below (clipped).

I went into mongo and ran a serverStatus command on my primary mongo node and only have about 300 connections going, so it's hardly working.

Below are my connection option settings in my server code:

auto_connect_retry = false    
connections_per_host = 10     
threads_multiplier = 10       
max_wait_time = 120000       
connect_timeout = 10000       
socket_timeout = 0           

Do I have something mis-configured?

Sep 9, 2013 8:31:26 PM com.mongodb.DBPortPool gotError
WARNING: emptying DBPortPool to /10.0.8.10:27017 b/c of error
java.net.SocketException: Connection timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:146)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at org.bson.io.Bits.readFully(Bits.java:46)
    at org.bson.io.Bits.readFully(Bits.java:33)
    at org.bson.io.Bits.readFully(Bits.java:28)
    at com.mongodb.Response.<init>(Response.java:40)
    at com.mongodb.DBPort.go(DBPort.java:142)
    at com.mongodb.DBPort.call(DBPort.java:92)
    at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:244)
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:288)
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:273)
    at com.mongodb.DBCollection.findOne(DBCollection.java:347)
    at com.mongodb.DBCollection.findOne(DBCollection.java:332)
    at com.mongodb.casbah.MongoCollectionBase$class.findOneByID(MongoCollection.scala:232)
    at com.mongodb.casbah.MongoCollection.findOneByID(MongoCollection.scala:866)
    at com.novus.salat.dao.SalatDAO.findOneById(SalatDAO.scala:353)
    at com.novus.salat.dao.ModelCompanion$class.findOneById(ModelCompanion.scala:173)

Answer 1

Generally a connection timeout occurs from one of the following in a replica set
1) All members are not able to communicate with each other
2) A program is connecting to replica for update and it is unable to send it to primary due to overload or 1st as well
3) All relicas are not in sync and one is lagging behind too much
4) Leader election is going on but not completed due to some reason

Please check if your relica set is consistent and all nodes are working by issuing rs.status() on primary node , also as earlier suggested check primary logs for more information