Poor performance on bulk deleting a large collection mongodb

1k Views Asked by At

I have a single standalone mongo installation on a Linux machine. The database contains a collection with 181 million documents. This collection is by far the largest collection in the database (approx 90%) The size of the collection is currently 3.5 TB. I'm running Mongo version 4.0.10 (Wired Tiger)

The collection have 2 indexes.

  • One on id
  • One on 2 fields and it is used when deleting documents (see those in the snippet below).

When benchmarking bulk deletion on this collection we used the following snippet

db.getCollection('Image').deleteMany(
{$and: [
{"CameraId" : 1}, 
{"SequenceNumber" : { $lt: 153000000 }}]})

To see the state of the deletion operation I ran a simple test of deleting 1000 documents while looking at the operation using currentOp(). It shows the following.

"command" : {
                "q" : {
                    "$and" : [ 
                        {
                            "CameraId" : 1.0
                        }, 
                        {
                            "SequenceNumber" : {
                                "$lt" : 153040000.0
                            }
                        }
                    ]
                },
                "limit" : 0
            },
            "planSummary" : "IXSCAN { CameraId: 1, SequenceNumber: 1 }",
            "numYields" : 876,
            "locks" : {
                "Global" : "w",
                "Database" : "w",
                "Collection" : "w"
            },
            "waitingForLock" : false,
            "lockStats" : {
                "Global" : {
                    "acquireCount" : {
                        "r" : NumberLong(877),
                        "w" : NumberLong(877)
                    }
                },
                "Database" : {
                    "acquireCount" : {
                        "w" : NumberLong(877)
                    }
                },
                "Collection" : {
                    "acquireCount" : {
                        "w" : NumberLong(877)
                    }
                }
            }

It seems to be using the correct index but the number and type of locks worries me. As I interpret this it aquires 1 global lock for each deleted document from a single collection.

When using this approach it has taken over a week to delete 40 million documents. This cannot be expected performance.

I realise there other design exists such as bulking documents into larger chunks and store them using GridFs, but the current design is what it is and I want to make sure that what I see is expected before changing my design or restructuring the data or even considering clustering etc.

Any suggestions of how to increase performance on bulk deletions or is this expected?

0

There are 0 best solutions below