Deleted data reappearance issue

31 Views Asked by At

I'm trying to reproduce a deleted data reappearance issue with a 2 node cluster, RF=2. I did with follow steps:

  1. alter table gc_grace_second to 180s.
  2. count table's rows, 8 rows
  3. shutdown node2
  4. delete 4 rows at node1
  5. wait 180s then compact the table
  6. restart node2 and count table's rows, 8 rows; count tables row at node1: 4 rows.
  7. nodetool repair -full
  8. check rows count at each node, both have 4 rows.

So after step 8, I'm confused since the official doc says: enter image description here

So my expectation was that the deleted data will appear after node2 is restarted. But it's disappear after 'repair -full' obviously.

BUT just one hour later, when I recheck the table: count() result is 8 at node1 and 0 at node2. Then I repair -full table again, and 2 nodes have identical count() result: 8. So deleted data just reappeared.

My question is:

  1. Can this test considered as a successful reproduction of deleted data reappearance?
  2. Why deleted data didn't reappear immediately right after node2 restart. I mean yeah, it's 8 rows after restart but it changed to 4 rows after repair full. My expectation was node1 count() increasing to 8 rows rather than node1 count() decline to 4 rows.
  3. Why data reappear one hour ago at last?
0

There are 0 best solutions below