Unable to restore elasticsearch snapshots from S3 using the repository-s3 plugin after nodes data directory got deleted from all nodes

Question

I am using repository-s3 plugin for snapshot and restore with elasticsearch 7.5.1.
I created snapshot policies and took snapshot of specific indices, confirmed that they existed in my S3 bucket. Now due to some reason, I had to delete data from all my nodes manually, so I ran
rm -r /var/lib/elasticsearch/nodes/0/ for all the nodes in my cluster.

Now when I again go to the snapshot and restore tab in kibana, it doesn't show my old snapshots data and I am not able to restore my indexes even though they are present in my S3 bucket. I need to restore the indexes and need help with the same.

Answer 1

By removing /var/lib/elasticsearch/nodes/0/ you also deleted your cluster state (i.e. the _state sub-folder that sits next to the indices sub-folder) which also happens to contain your repository definitions.

You should never delete data directly from the filesystem unless you know what you're doing.

If you need space, just DELETE * from Dev Tools, but don't venture into the file system.

What you can try to do now is to recreate your S3 repository anew with the exact same settings as before, that will restore the repository in your cluster state and you might be able to see your old snapshots.