Mongo performance

Question

Our test tool creates n threads and execute m iteration inside each thread. We calculate min, avg and max operation time. So that we see from time to time mongo executes request extremely slow - up to few seconds. Also each time first operation are slowest, we even exclude it intentionally from statistics and still difference between avg and max is huge. Is it normal? Can I eliminate those slow operations? Why mongo executes some of requests so extremely slow?

Below results for single operations mode and mix mode when we run simultaneously tests for insert/get/remove/update. In mix mode thread=1 means we created 4 threads one for each type of test

We used default value - 100 connections per host

Source

public void storeMt(MyTestObject myTestObject) {
    mongoTemplate.insert(myTestObject );
  }

  public MyTestObject getMt(long id) {
    MyTestObject result = mongoTemplate.findById(id, MyTestObject.class);
    return result;
  }


@Document
public class MyTestObject implements Serializable {

  private static final long serialVersionUID = 1L;
  @Id
  private long id;
//class contains 20+ fields
.......

Answer 1

I see few reasons why this may be happening.

1. It is not due to mongoDB it may be due to the java process. It takes time in the beginning to start and load(allocate memory) relevant variables in memory.
2. Use MMS, a monitoring tool for MongoDB to see if the database is a bottleneck. You can check the locks on database, network usage, memory usage.
3. MongoDB acquires lock on database level for the current version. And so if your query is hitting the same database (different or same collection of a single database), it may slow down the operation.
4. Also, adding more number of threads not necessarily increase the performance. It may bring down the performance due to extra load on CPU,Memory(RAM) etc. So, I suggest increase number of thread after frequent interval and then see when the performance stops improving.

I am sure if you do this, you will find out the exact reason why it is slowing down.

Cheers!

Answer 2

I think this is an expected behavior, whenever a query is executed, the (winning) query plan is cached for further retrieval.

That query cache is also getting dumped time and again so occasionally it takes longer.

In your case some query seems costly, I think you need to identify those queries and take measures to enhance the performance, If it's a read query, maybe a (compound) index is needed.

If it's a write query, it would need further analysis, what sort of collection it is? what indexes it has and what's the count of documents?

Here is more on query plan caching