How does Elasticsearch incremental snapshots deal with the deleted docs?

Question

I regularly take a snapshot of my ES cluster in a s3 bucket and so i wanted to know that if i am deleting my old docs from the cluster and regularly adding new docs then after taking a snapshot how does ES deal with this scenario wheather the docs get deleted from my previous snapshots as well or how does ES keep a backup of my docs. Please explain?

Answer 1

When ES takes snapshots, ES doesn't take snapshots of docs but rather it takes snapshots of segments. Of course, the segments contain the docs.

To understand the concept of incremental, let's take the below example.

Say there's an index called my_index with 1 primary shard (shard 0). As data gets written to the index, it will create segment file(s) for the shards.

Initially, the index my_index may look like:

"my_index"
"consists of shard 0"
"shard 0 consists of segements A,B,C"

You take Snapshot S1 of index my_index at time T1.

The snapshot S1 contains the following metadata:

Index: my_index
Shards: 0
Segments: A,B,C
And then it will copy the segment files.

Now, you index more data. ES merges segements B and C into a new segment D and adds new segment E for new data. Once segments are merged, the old segments are deleted from the shard. Same way, when documents are deleted, segment merging happens

Now the shard 0 of index my_index contains segments A,D,E

You take Snapshot S2 of index my_index at time T2. S2 will check to see what files it will need.

It will NOT copy segment A (because it already exists in the repo - this is what is meant by incremental). 
It will copy segment D
it will copy segment E

The snapshot S2 contains the following metadata:

Index: my_index Shards: 0 Segments: A,D,E

What is incremental here? The incremental nature is for new segment files not necessarily for new data. For Snapshot S2, segment A was NOT copied because it was already contained in S1.

When happens when you delete Snapshot S1?
1. Segments B and C will be deleted since they are no longer being referenced
2. Exclude Segment A since it's being referenced by Snapshot S2

When happens when you delete index my_index?
The snapshots will still contain segement files pertaining to my_index allowing you to recover the index anytime.

What happens when documents are deleted? When docs are deleted, eventually the segment files are merged, new segments are created. So when you take a snapshot after document has been deleted, the snapshot will not have the document.

Hope this helps