Mongodb IndexBuildRetry keep rebooting server (noIndexBuildRetry)

Question

I have a mongo DB that refuse to restart because of a unique index problem. Since the DB doesn't start, its stuck on this reboot loop and I can't connect to it to remove the index (or clean the buggy data).

I'd happily fix the unique key/index problem, but it's not possible to use commands like db.coll.dropIndexes(). What would be the best way to resolve this issue?

option 1 - remove indexes from docker filesystem

Is there another way to remove indexes from the file system directly? (I'm able to connect to the docker for a few seconds to run commands).

option 2 - set the noIndexBuildRetry flag using dokku

I'm using Dokku, which doesn't seem to offer an easy way to pass on the suggested --noIndexBuildRetry flag to the mongod process.

This article mention the /etc/init.d/mongod config file, but it doesn't exist on my docker image. How can this flagged be set in a Dokku infrastructure?

option 3

Update to mongo 4.x. It seems to handle Indexes differently and could be more permissive with my duplicate key (and not prevent the server to start).

Mongo server logs:

2020-12-16T14:12:00.159+0000 E INDEX    [initandlisten] Index rebuilding did not complete: DuplicateKey: E11000 duplicate key error collection: db1.Events index: ns_1_hash_1 dup key: { : "userhash", : null }
2020-12-16T14:12:00.159+0000 I INDEX    [initandlisten] note: restart the server with --noIndexBuildRetry to skip index rebuilds
2020-12-16T14:12:00.160+0000 F -        [initandlisten] Fatal Assertion 26100 at src/mongo/db/index_rebuilder.cpp 136

Answer 1

Use docker run to run a shell (e.g. bash) in the container instead of using the default entry point which I assume launches some combination of mongod/mongos. Once in the shell, start the database manually using any options you like.

Answer 2

It's possible to customize the mongod flags in dokku-mongo using a MONGO_CONFIG_OPTIONS file. As of now, you'll need to restart the container (not just the service) after editing this file.

Answer 3

If you are using centos you have to check the systemd conf of mongodb. In that you have to change the line "OPTIONS". In that line add "--noIndexBuildRetry" at the end like below,

Environment="OPTIONS=-f /etc/mongod.conf --noIndexBuildRetry"

Then reload your systemctl by,

sytemctl daemon-reload

And then restart your mongodb,

systemctl restart mongodb