I'm using xtdb in a testing environment with a RocksDB backend. All was well until yesterday, when the system stopped ingesting new data. It tells me that this is because of "missing docs", and gives me the id of the allegedly missing doc, but since it is missing, that doesn't tell me much. I have a specific format for my xt/ids (basically type+guid) and this doesn't match that format, so I don't think this id is one of mine. Calling history on the entity id just gives me an empty vector. I understand the block on updates for consistency reasons, but how to diagnose and recover from this situation (short of trashing the database and starting again)? This would obviously be a massive worry were it to happen in production.
How to recover from "missing docs" in xtdb?
62 Views Asked by Kaz Librowski At
1
In the general case this "missing docs" error indicates a corrupted document store and the only proper resolution is to manually restore/recover based on a backup of the document store. This almost certainly implies some level of data loss.
However, there was a known bug in the transaction function logic prior to 1.22.0 which could intermittently produce this error (but without any genuine data loss), see https://github.com/xtdb/xtdb/commit/1c30550fb14bd6d09027ff902cb00021bd6e57c4
However, if you weren't using transaction functions then there may be another unknown explanation.