MongoDB Unrecoverable Failure

Question

My Mac (OSX 9) had a power blip and when restarted, I have one db in Mongo that is completely unrecoverable. I am using the db per directory with dbpath and logpath specified options and was able to recover all except one of the dbs. This is the log from the unrecoverable one. Any ideas would be helpful - I have run --repair and removed the lock file with zero results.

LOG FILE:

2014-12-01T09:40:40.607-0700 [initandlisten] MongoDB starting : pid=1273 port=27017 dbpath=/mongo/dbs/ 64-bit host=bbcoms-imac-2.body.local
2014-12-01T09:40:40.607-0700 [initandlisten]
2014-12-01T09:40:40.607-0700 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
2014-12-01T09:40:40.608-0700 [initandlisten] db version v2.6.5
2014-12-01T09:40:40.608-0700 [initandlisten] git version: nogitversion
2014-12-01T09:40:40.608-0700 [initandlisten] build info: Darwin minimavericks.local 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2014-12-01T09:40:40.608-0700 [initandlisten] allocator: tcmalloc
2014-12-01T09:40:40.608-0700 [initandlisten] options: { repair: true, storage: { dbPath: "/mongo/dbs/", repairPath: "/mongo/db" } }
2014-12-01T09:40:40.608-0700 [initandlisten] exception in initAndListen: 12590 repairpath (/mongo/db) does not exist, terminating
2014-12-01T09:40:40.608-0700 [initandlisten] dbexit:
2014-12-01T09:40:40.608-0700 [initandlisten] shutdown: going to close listening sockets...
2014-12-01T09:40:40.608-0700 [initandlisten] shutdown: going to flush diaglog...
2014-12-01T09:40:40.609-0700 [initandlisten] shutdown: going to close sockets...
2014-12-01T09:20:58.339-0700 [clientcursormon]  mapped (incl journal view):2176
2014-12-01T09:20:58.339-0700 [clientcursormon]  connections:1
2014-12-01T09:25:58.390-0700 [clientcursormon] mem (MB) res:41 virt:4655
2014-12-01T09:25:58.390-0700 [clientcursormon]  mapped (incl journal view):2176
2014-12-01T09:25:58.390-0700 [clientcursormon]  connections:1
2014-12-01T09:28:53.518-0700 [conn1] end connection 127.0.0.1:50225 (0 connections now open)
2014-12-01T09:30:55.153-0700 [initandlisten] connection accepted from 127.0.0.1:50411 #3 (1 connection now open)
2014-12-01T09:30:57.586-0700 [conn3] command admin.$cmd command: listDatabases { listDatabases: 1.0 } keyUpdates:0 numYields:0 locks(micros) R:3 W:105826 r:14 reslen:337 106ms
2014-12-01T09:30:58.313-0700 [TTLMonitor] articles.system.indexes Assertion failure isOk() src/mongo/db/storage/extent.h 80
2014-12-01T09:30:58.316-0700 [TTLMonitor] articles.system.indexes 0x1010665aa 0x101011135 0x100ffdbf2 0x100e1ca0c 0x100e4d4bc 0x100977b13 0x100ad7187 0x100cfdbcb 0x100ce5e6d 0x100b7d0b7 0x100b85055 0x100b852bb 0x1008dae1b 0x1008bee8a 0x100b854f1 0x100e50be6 0x100e4fe07 0x101000606 0x10109f2a1 0x7fff84fd0899
 0   mongod                              0x00000001010665aa _ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE + 58
 1   mongod                              0x0000000101011135 _ZN5mongo10logContextEPKc + 453
 2   mongod                              0x0000000100ffdbf2 _ZN5mongo12verifyFailedEPKcS1_j + 626
 3   mongod                              0x0000000100e1ca0c _ZNK5mongo13ExtentManager9getExtentERKNS_7DiskLocEb + 108
 4   mongod                              0x0000000100e4d4bc _ZN5mongo12FlatIteratorC2EPKNS_10CollectionERKNS_7DiskLocERKNS_20CollectionScanParams9DirectionE + 92
 5   mongod                              0x0000000100977b13 _ZNK5mongo10Collection11getIteratorERKNS_7DiskLocEbRKNS_20CollectionScanParams9DirectionE + 115
 6   mongod                              0x0000000100ad7187 _ZN5mongo14CollectionScan4workEPm + 519
 7   mongod                              0x0000000100cfdbcb _ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_7DiskLocE + 283
 8   mongod                              0x0000000100ce5e6d _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3101
 9   mongod                              0x0000000100b7d0b7 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 2663
 10  mongod                              0x0000000100b85055 _ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPNSt3__112basic_stringIcNS3_11char_traitsIcEENS3_9allocatorIcEEEE + 165
2014-12-01T09:40:40.609-0700 [initandlisten] shutdown: waiting for fs preallocator...
2014-12-01T09:40:40.609-0700 [initandlisten] shutdown: closing all files...
2014-12-01T09:40:40.610-0700 [initandlisten] closeAllFiles() finished
2014-12-01T09:40:40.610-0700 [initandlisten] dbexit: really exiting now

Answer 1

I see the error about an assertion failure with an extent, but above that you have the lines

2014-12-01T09:40:40.608-0700 [initandlisten] options: { repair: true, storage: { dbPath: "/mongo/dbs/", repairPath: "/mongo/db" } }
2014-12-01T09:40:40.608-0700 [initandlisten] exception in initAndListen: 12590 repairpath (/mongo/db) does not exist, terminating

so it seems you aren't supplying a valid repairPath when you are running repair. Can you start the server? Is this mongod part of a replica set? If the server can start, can you run db.collection.validate on the collections in the bad database? What happens?