AWS opensearch: Cannot update advanced option `override_main_response_version`

Question

I have an AWS opensearch with vanilla settings. I needed to change the advanced options to enable override_main_response_version=true so that filebeat could work. But whichever way I try, it just does not work. Any workarounds?

Method 1: terraform:

resource "aws_elasticsearch_domain" "this" {
...
  advanced_options = {
    "override_main_response_version" = "true"
  }
...
}

This results in

Unrecognized advanced option 'override_main_response_version' passed in advancedOptions

Method 2: awscliv2

aws opensearch update-domain-config --advanced-options 'override_main_response_version=true' --domain-name whatever

which results in:

An error occurred (ValidationException) when calling the UpdateDomainConfig operation: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.

I don't see any way on the AWS console to change this option. Please help!

Answer 1

This option can be enabled from opensearch's dev tools:

PUT /_cluster/settings
{
"persistent" : {
"compatibility.override_main_response_version" : true
}
}

Answer 2

Currently, this is not possible via Terraform, you need to query the cluster as @ixaxaar wrote. I noticed that the newer versions of OpenSearch have this setting enabled by default. I disabled this in my cluster (set it to false):

Context: Why is this setting even present?

Agents and ingestion tools

Historically, many multiple popular agents and ingestion tools have worked with Elasticsearch OSS, such as Beats, Logstash, Fluentd, FluentBit, and OpenTelemetry. OpenSearch aims to continue to support a broad set of agents and ingestion tools, but not all have been tested or have explicitly added OpenSearch compatibility.

As an intermediate compatibility solution, OpenSearch has a setting that instructs the cluster to return version 7.10.2 rather than its actual version.

If you use clients that include a version check, such as versions of Logstash OSS or Filebeat OSS between 7.x - 7.12.x, enable the setting:

PUT _cluster/settings {
"persistent": { "compatibility": { "override_main_response_version": true }
} }

Source: https://opensearch.org/docs/latest/tools/index/#agents-and-ingestion-tools