Aerospike 3.7.4 after reload there is a data loss and some deleted data was restored automatically

Question

We have a cluster from 3 nodes. I have a namespace to store the history of operations, I did lots of delete operations on one of the set and after that migrated data from scratch.
For some reason after a while one node has failed, and we needed to reload the cluster, and later we have encountered that most of the new data was lost and some deleted data was restored.
Can you please help to avoid such behaviour as we need consistency.

Aerospike version is 3.7.4
Here is the configuration for the namespace:

namespace dar_history {
   replication-factor 2
   memory-size 4G
   default-ttl 0 # 30 days, use 0 to never expire/evict.

   storage-engine device {
           file ../dar_history.dat
           filesize 32G
           data-in-memory true # Store data in memory in addition to file.
    }
}

Answer 1

Aerospike 3.10.0 introduces durable deletes for Aerospike Enterprise. Learn more about how they work here: https://www.aerospike.com/docs/guide/durable_deletes.html.

A few alternative solutions for community edition are discussed here: https://discuss.aerospike.com/t/expired-deleted-data-reappears-after-server-is-restarted/470/22.

Answer 2

We have solved the problem by backing up the data, then restarting the cluster, importing data from the backup, recreating all the indices we had and deleting old unnecessary data. Thank you all for replies.