MongoDB not using multikey index

Question

I have a MongoDB version 3.0.3 collection which contains documents with 2 fields:

• A numeric long id
• An array of 60 numeric Doubles.

I built a multikey index on the array and confirmed that the index exists and is multikey by using the getIndexInfo() method. However, when I query on the intersection of 2 ranges of specific fields in the array, Mongo does not use this index, despite the fact that there are 1,000,000 documents in the collection. I can see this from the output of the explain() method. Even stranger, when I specify use of the index with a hint, Mongo traverses all 1,000,000 documents and 60,000,000 index entries, which I see in the output of explain().

I am constructing the query with the following code:

BasicDBObject q2 = new BasicDBObject("array.0",new BasicDBObject("$lt",1000.0));
BasicDBObject q1 = new BasicDBObject("array.1",new BasicDBObject("$gte",800.0));
BasicDBObject q_and = new BasicDBObject("$and",Arrays.asList(q1,q2));
dbo = collection.find(q_and).explain();

Any thoughts?

Thanks in advance for your help.

In reply to the request for explain output:

Without specifying a hint, the explain output is as follows:

{ "queryPlanner" : 
{ "plannerVersion" : 1 , "namespace" : "local.TestArrays" , "indexFilterSet" : false , "parsedQuery" : 
{ "$and" : [ 
{ "array.0" : 
{ "$lt" : 1000.0
}
} , 
{ "array.1" : 
{ "$gte" : 800.0
}
}]
} , "winningPlan" : 
{ "stage" : "COLLSCAN" , "filter" : 
{ "$and" : [ 
{ "array.0" : 
{ "$lt" : 1000.0
}
} , 
{ "array.1" : 
{ "$gte" : 800.0
}
}]
} , "direction" : "forward"
} , "rejectedPlans" : [ ]
} , "executionStats" : 
{ "executionSuccess" : true , "nReturned" : 2 , "executionTimeMillis" : 2248 , "totalKeysExamined" : 0 , "totalDocsExamined" : 1000000 , "executionStages" : 
{ "stage" : "COLLSCAN" , "filter" : 
{ "$and" : [ 
{ "array.0" : 
{ "$lt" : 1000.0
}
} , 
{ "array.1" : 
{ "$gte" : 800.0
}
}]
} , "nReturned" : 2 , "executionTimeMillisEstimate" : 2190 , "works" : 1000002 , "advanced" : 2 , "needTime" : 999999 , "needFetch" : 0 , "saveState" : 7812 , "restoreState" : 7812 , "isEOF" : 1 , "invalidates" : 0 , "direction" : "forward" , "docsExamined" : 1000000
} , "allPlansExecution" : [ ]
} , "serverInfo" : 
{ "host" : "NYDEVWS0005052" , "port" : 27017 , "version" : "3.0.3" , "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105"
}
}

When specifying an index to use with the following code:

collection.find(q_and).hint("array_1").explain();

the explain output is as follows:

{ "queryPlanner" : 
{ "plannerVersion" : 1 , "namespace" : "local.TestArrays" , "indexFilterSet" : false , "parsedQuery" : 
{ "$and" : [ 
{ "array.0" : 
{ "$lt" : 1000.0
}
} , 
{ "array.1" : 
{ "$gte" : 800.0
}
}]
} , "winningPlan" : 
{ "stage" : "KEEP_MUTATIONS" , "inputStage" : 
{ "stage" : "FETCH" , "filter" : 
{ "$and" : [ 
{ "array.0" : 
{ "$lt" : 1000.0
}
} , 
{ "array.1" : 
{ "$gte" : 800.0
}
}]
} , "inputStage" : 
{ "stage" : "IXSCAN" , "keyPattern" : 
{ "array" : 1
} , "indexName" : "array_1" , "isMultiKey" : true , "direction" : "forward" , "indexBounds" : 
{ "array" : [ "[MinKey, MaxKey]"]
}
}
}
} , "rejectedPlans" : [ ]
} , "executionStats" : 
{ "executionSuccess" : true , "nReturned" : 2 , "executionTimeMillis" : 61401 , "totalKeysExamined" : 60000000 , "totalDocsExamined" : 1000000 , "executionStages" : 
{ "stage" : "KEEP_MUTATIONS" , "nReturned" : 2 , "executionTimeMillisEstimate" : 56570 , "works" : 60001744 , "advanced" : 2 , "needTime" : 59999998 , "needFetch" : 1743 , "saveState" : 470130 , "restoreState" : 470130 , "isEOF" : 1 , "invalidates" : 0 , "inputStage" : 
{ "stage" : "FETCH" , "filter" : 
{ "$and" : [ 
{ "array.0" : 
{ "$lt" : 1000.0
}
} , 
{ "array.1" : 
{ "$gte" : 800.0
}
}]
} , "nReturned" : 2 , "executionTimeMillisEstimate" : 55620 , "works" : 60001744 , "advanced" : 2 , "needTime" : 59999998 , "needFetch" : 1743 , "saveState" : 470130 , "restoreState" : 470130 , "isEOF" : 1 , "invalidates" : 0 , "docsExamined" : 1000000 , "alreadyHasObj" : 0 , "inputStage" : 
{ "stage" : "IXSCAN" , "nReturned" : 1000000 , "executionTimeMillisEstimate" : 50820 , "works" : 60000000 , "advanced" : 1000000 , "needTime" : 59000000 , "needFetch" : 0 , "saveState" : 470130 , "restoreState" : 470130 , "isEOF" : 1 , "invalidates" : 0 , "keyPattern" : 
{ "array" : 1
} , "indexName" : "array_1" , "isMultiKey" : true , "direction" : "forward" , "indexBounds" : 
{ "array" : [ "[MinKey, MaxKey]"]
} , "keysExamined" : 60000000 , "dupsTested" : 60000000 , "dupsDropped" : 59000000 , "seenInvalidated" : 0 , "matchTested" : 0
}
}
} , "allPlansExecution" : [ ]
} , "serverInfo" : 
{ "host" : "NYDEVWS0005052" , "port" : 27017 , "version" : "3.0.3" , "gitVersion" : "b40106b36eecd1b4407eb1ad1af6bc60593c6105"
}
}

Answer 1

I am afraid your problem is that you are querying specific elements of an array. So even if you have an index on that array, the query must explicitly scan the whole collection, fetch the elements 0 and 1 from the array and execute matching.

If you need to run your matching exactly on specific elements, it is worth pulling them out of the array and have them as separate fields, and then create indexes on them. So consider this:

{
   ...
   array: [...],
   ex_element_0: "value0",
   ex_element_1: "value0",
   ...
}

Answer 2

I misunderstood the meaning of multikey indexes. I thought a separate index was created for each position in the array. However, upon further reading I see that a SINGLE index is created for the array and all entries for a document are entered into that single index, not preserving array positions. I took n9code's advice and created separate fields and an index on each field. That works as expected.