Curator not deleting older index

Question

I have an index named 'locations' where I pushed some data few days ago. I want to delete all data older than 1 day. My config looks like -

actions: 1:

action: delete_indices

description: >-

  Delete indices older than 10 days (based on index name), for locations
  prefixed indices.

options:
  ignore_empty_list: True
  disable_action: True
filters:
- filtertype: pattern
  kind: prefix
  value: locations
- filtertype: age
  source: creation_date
  direction: older
  unit: days
  unit_count: 1
- filtertype: count
  count: 1
options: 
  disable_action: false
  ignore_empty_list: true
  allow_ilm_indices: true

---

However, when I run this config I get below -

2021-04-30 03:33:35,639 DEBUG curator.indexlist iterate_filters:1244 Pre-instance: ['locations']

2021-04-30 03:33:35,639 DEBUG curator.indexlist filter_by_count:928 Filtering indices by count

2021-04-30 03:33:35,639 DEBUG curator.indexlist working_list:237 Generating working list of indices

2021-04-30 03:33:35,639 DEBUG curator.indexlist __not_actionable:38 Index locations is not actionable, removing from list.

2021-04-30 03:33:35,655 DEBUG curator.indexlist __excludify:58 Removed from actionable list: locations is 1 of specified count of 1.

2021-04-30 03:33:35,655 DEBUG curator.indexlist iterate_filters:1246 Post-instance: []

2021-04-30 03:33:35,655 DEBUG curator.actions.delete_indices init:612 master_timeout value: 30s

2021-04-30 03:33:35,655 DEBUG curator.cli process_action:103 Doing the action here.

2021-04-30 03:33:35,655 DEBUG curator.indexlist empty_list_check:226 Checking for empty list

2021-04-30 03:33:35,655 INFO curator.cli run:202 Skipping action "delete_indices" due to empty list: <class 'curator.exceptions.NoIndices'>

2021-04-30 03:33:35,655 INFO curator.cli run:225 Action ID: 1, "delete_indices" completed.

2021-04-30 03:33:35,655 INFO curator.cli run:226 Job completed.

---

What configuration am I missing ?

Answer 1

Curator manages whole indices only. Curator either deletes an index, or it doesn't. It cannot delete data from within an index.

However, one potential way to accomplish what you seek would be to use Elasticsearch's delete_by_query API. However, I strongly discourage this if you are indexing a continuous stream of time-based data (e.g. logs, metrics, etc.). It is much more efficient to use rollover indices and delete the old indices as a whole. Think of the difference in impact to your Elasticsearch cluster as being the same difference between a SQL DROP TABLE vs. a DELETE FROM TABLE WHERE... statement. In the former, a single statement deletes the entire table as a single operation, but in the latter it could be tens of thousands of individual operations. A delete_by_query is similar in Elasticsearch terms, and there are other negative impact reasons in pursuing that if it can be avoided.