Bitnami Mongodb ReplicaSet Set Issue on MS Azure - Secondary node late response

Question

I have created Mongodb with Replica Set on Azure. I have rails application hitting replica set on public ip. My replica set have 1 primary and 2 secondary node. I am facing extreme lag in response from secondary nodes but very fast response from primary node for isMaster request.

Query taking to much time for secondary node

COMMAND database=admin command={:ismaster=>1}

Mongoid 4.0.1 Rails 4.2.5

Why mongodb calling to secondary node? I am ok if it is calling but why it is taking so much time which is holding my application request for around 12 seconds (6 seconds for each secondary node).

I appreciate your help in advance.

Sharing application log as below

I, [2016-11-30T11:27:05.351584 #11924] INFO -- : Started GET "/" for 175.100.138.183 at 2016-11-30 11:27:05 +0000 I, [2016-11-30T11:27:05.352680 #11924] INFO -- : Processing by WelcomeController#index as HTML I, [2016-11-30T11:27:05.352786 #11924] INFO -- : Parameters: {"request_client"=>"production_abc_io", "client_database"=>"localhost"}

// This call to primary node return quick reponse D, [2016-11-30T11:27:05.356846 #11924] DEBUG -- : MOPED: 10.0.0.4:27017 COMMAND database=admin command={:ismaster=>1} runtime: 0.6903ms

//This call to secondary node taking around 6 seconds D, [2016-11-30T11:27:11.356397 #11924] DEBUG -- : MOPED: 10.0.0.5:27017 COMMAND database=admin command={:ismaster=>1} runtime: 5999.3523ms

//This call to another secondary node taking around 6 seconds D, [2016-11-30T11:27:17.356509 #11924] DEBUG -- : MOPED: 10.0.0.6:27017 COMMAND database=admin command={:ismaster=>1} runtime: 5999.8489ms

D, [2016-11-30T11:27:17.357908 #11924] DEBUG -- : MOPED: replica_set_public_ip:27017 QUERY database=localhost collection=clients selector={"_id"=>BSON::ObjectId('55e9a684747265a004000000')} flags=[:slave_ok] limit=0 skip=0 batch_size=nil fields=nil runtime: 1.1453ms

D, [2016-11-30T11:27:17.360061 #11924] DEBUG -- : MOPED: replica_set_public_ip:27017 QUERY database=localhost collection=base_users selector={"$query"=>{"client_

Answer 1

By default, all the read operation are routed to the primary node unless you had changed the read preference parameter. This parameter could explain why the primary node is calling to the secondary ones.

In order to figure out why the queries to the secondaries are slow you can take a look to the mongodb logs located at '/opt/bitnami/mongodb/logs/mongodb.log'. Maybe there is some information there.

Besides, you can use the mongotop command to monitor basic usage statistics for each collection and the mongostats command to monitor basic MongoDB server statistics.