Why is Elasticsearch-curator `create_index` dry-run successful but then actual run isn't?

Question

I'm learning how to work with Elasticsearch and am running into a problem where my create_index keeps failing. It appears to work correctly on the dry-run but then fails on the actual run. I'm not sure what I'm doing wrong, I checked google but everybody else doesn't seem to have a similar issue. Thanks for you help.

here's my ACTION.YML:

actions:
  1:
    action: create_index
    description: "create the new index"
    options:
      name: creating_some_metrics.v1
      extra_settings:
        number_of_replicas: 4
        number_of_shards: 4
        refresh_interval: 1s
        mappings:
          parquet-metrics:
            properties:
              lists:
                properties:
                  success_date_history:
                    type: date
                  process_time_history:
                    type: date
                  stat_delta_seconds_history:
                    type: integer
                  options:
                    type: text
              '@timestamp':
                type: date
              id_schema:
                type: integer
              bucket:
                type: keyword
                fields:
                  text:
                    type: text
              error_code:
                type: keyword
              error_reason:
                type: text
                fields:
                  keyword:
                    type: keyword
                    ignore_above: 256
              error_trace:
                type: text
                fields:
                  keyword:
                    type: keyword
                    ignore_above: 256
              success_date:
                type: date
              object_key:
                type: text
                fields:
                  keyword:
                    type: keyword
                    ignore_above: 256
              object_path:
                type: text
                fields:
                  keyword:
                    type: keyword
                    ignore_above: 256
              partition_time:
                type: date
              partition_time_str:
                type: text
              process_time:
                type: date
              stat_delta_seconds:
                type: integer
              stat_file_count:
                type: integer
              stat_row_count:
                type: long
              stat_total_size:
                type: long
              stat_type:
                type: keyword
              status:
                type: keyword
              tag_id:
                type: text
                fields:
                  keyword:
                    type: keyword
                    ignore_above: 256

When I execute curator --config CONFIG.YML --dry-run ACTION.YML I get:

2021-08-25 14:09:06,083 INFO                 curator.cli                    run:225  Action ID: 1, "create_index" completed.
2021-08-25 14:09:06,083 INFO                 curator.cli                    run:226  Job completed.

When I execute curator --config CONFIG.YML ACTION.YML I run into this issue:

Failed to complete action: create_index.  <class 'curator.exceptions.FailedExecution'>: Exception encountered.
Exception: TransportError(500, '{"status:":"INTERNAL_SERVER_ERROR","message":"Internal server error"})

What am I doing wrong here?

Answer 1

It's impossible to do a true dry run with this particular action as Elasticsearch does not have a concept of a dry-run for index creation. All that Curator can here do is validate both the YAML formatting (not the API call itself) and that it can connect to Elasticsearch. The --dry-run flag here really only does a no-op so that it doesn't cause a problem.

You're receiving a 500 error back from Elasticsearch, which means something is going on upstream in the Elasticsearch server. You may be able to see more by looking in the Elasticsearch logs directly, and/or set:

logging:
  loglevel: DEBUG
  blacklist: []

in your CONFIG.YML file. This will un-hide the full Elasticsearch server responses, which are usually very verbose.