Manage replicas of ISM indices with templates

Question

I have a standalone instance with Opensearch for testing purposes, I want to keep it light and clean so I'm using ISM to delete indices older than x days.

What I noticed is that by default Opensearch generates a management index (".opensearch-ism-config") with replica "1".

Since I'm using a standalone instance (it is just testing, I'm not worried with redundancy, HA or anything like that) and want to keep my cluster with green status, I have decided that I want those indices to have replica "0".

In order to achieve that, I have created a template in which I set replica "0" for these indices:

{
  "order" : 100,
  "version" : 1,
  "index_patterns" : [".opensearch-ism-*"],
  "settings" : {
    "index": {
      "number_of_shards" : "1",
      "number_of_replicas": 0
    }
  }
}

After a PUT, I start using ISM so that the management ISM index is created after this template is on Opensearch node.

What I observe is that all management indices from ISM are generated with replica "1", therefore ignoring the template.

I can set replica to "0" by updating index settings after creation but this is not the ideal scenario as ISM index rotate and new ones are generated from time to time.

Is there any way to have ISM indices applying replica "0" automatically ?

Answer 1

I ran into this same issue when transitioning from Elasticsearch 6 to OpenSearch. In Elasticsearch I was accustomed to index templates being additive. For example, in Elasticsearch 6, I could do something like this:

Create an index template to configure ILM rollover alias

PUT _template/a_descriptive_template_name-ilm
{
    "settings": {
        "index.lifecycle.name": "an_efficient_ilm_policy",
        "index.lifecycle.rollover_alias": "a_descriptive_index_name-ilm"
    },
    "order": 10,
    "index_patterns": [
        "a_descriptive_index_name-ilm-*"
    ]
}

Create an index template to configure index shard settings

PUT _template/a_common_template_name
{
    "settings": {
        "number_of_shards": 2,
        "number_of_replicas": 1
    },
    "order": 5,
    "index_patterns": [
        "a_descriptive_index_name-ilm-*"
    ]
}

Create the initial index

PUT a_descriptive_index_name-ilm-000001
{
    "aliases": {
        "a_descriptive_index_name-ilm": {
            "is_write_index": true
        }
    }
}

Verify the settings

GET a_descriptive_index_name-ilm-000001
{
  "a_descriptive_index_name-ilm-000001" : {
    "aliases" : {
      "a_descriptive_index_name-ilm" : {
        "is_write_index" : true
      }
    },
    "mappings" : { },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "an_efficient_ilm_policy",
          "rollover_alias" : "a_descriptive_index_name-ilm"
        },
        "number_of_shards" : "2",
        "provided_name" : "a_descriptive_index_name-ilm-000001",
        "creation_date" : "1692853100041",
        "number_of_replicas" : "1",
        "uuid" : "AJ3OiRaXRj2cVCvuKlUWgg",
        "version" : {
          "created" : "6081299"
        }
      }
    }
  }
}

When I tried to do this same technique in Opensearch 2.3, I noticed that it did not work as I expected. The template with the higher priority overwrote the settings in the lower priority template, completely, even if there were not overlapping settings. In the example below, where I am doing the same thing as Elasticsearch (but using different verbs for OpenSearch), you will see that the additive behavior is no longer supported:

Create an index template to configure ISM rollover alias

PUT _index_template/a_descriptive_template_name-ism
{
    "template": {
        "settings": {
            "plugins.index_state_management.rollover_alias": "a_descriptive_index_name-ism"
        }
    },
    "priority": 10,
    "index_patterns": [
        "a_descriptive_index_name-ism-*"
    ]
}

Create an index template to configure index shard settings

PUT _index_template/a_common_template_name
{
    "template": {
        "settings": {
          "number_of_shards": 2,
          "number_of_replicas": 1
        }
    },
    "priority": 5,
    "index_patterns": [
        "a_descriptive_index_name-ism-*"
    ]
}

Create the initial index

PUT a_descriptive_index_name-ism-000001
{
    "aliases": {
        "a_descriptive_index_name-ism": {
            "is_write_index": true
        }
    }
}

Verify the settings

{
  "a_descriptive_index_name-ism-000001" : {
    "aliases" : {
      "a_descriptive_index_name-ism" : {
        "is_write_index" : true
      }
    },
    "mappings" : { },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "a_descriptive_index_name-ism"
          }
        },
        "provided_name" : "a_descriptive_index_name-ism-000001",
        "creation_date" : "1692853523944",
        "number_of_replicas" : "1",
        "uuid" : "ZZSjabAVQACC7m8Xn0uKtA",
        "version" : {
          "created" : "136247827"
        }
      }
    }
  }
}

As you can see, the "number_of_shards": 2 setting gets overwritten.

To apply both the shard settings and the ISM rollover settings, (and whatever other settings / mappings required), there are two options.

1. Combine all the settings / mappings and ISM configuration into a single index template (or single index template per index pattern)
2. [Better Solution] Use the OpenSearch Composable index templates functionality to allow for "layering" templates.

The OP answered in a later comment "It does, I have another index template". To solve for this scenario, option 2 is the way I suggest to follow.

Here is an example:

Create a component template to configure index shard settings

PUT _component_template/shards_component_template
{
      "template": {
        "settings": {
          "number_of_shards": 2,
          "number_of_replicas": 1
        }
    }
}

Create the index template to configure ILM rollover alias, referencing the component template

PUT _index_template/a_descriptive_template_name-ism
{
    "template": {
        "settings": {
            "plugins.index_state_management.rollover_alias": "a_descriptive_index_name-ism"
        }
    },
    "priority": 10,
    "index_patterns": [
        "a_descriptive_index_name-ism-*"
    ],
    "composed_of": [
        "shards_component_template"
    ]
}

Create the initial index

PUT a_descriptive_index_name-ism-000001
{
    "aliases": {
        "a_descriptive_index_name-ism": {
            "is_write_index": true
        }
    }
}

Verify the settings

{
  "a_descriptive_index_name-ism-000001" : {
    "aliases" : {
      "a_descriptive_index_name-ism" : {
        "is_write_index" : true
      }
    },
    "mappings" : { },
    "settings" : {
      "index" : {
        "number_of_shards" : "2",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "a_descriptive_index_name-ism"
          }
        },
        "provided_name" : "a_descriptive_index_name-ism-000001",
        "creation_date" : "1692854275856",
        "number_of_replicas" : "1",
        "uuid" : "SGvVILFsSG-3PdpoCg8Tkw",
        "version" : {
          "created" : "136247827"
        }
      }
    }
  }
}