Opensearch issues with json field names containing []

Question

here is a sample object that comes into os logstash

"MyObject": {
  "normalField": "asdf",
  "additionalData:long:subObject[0]": "banking",
  "additionalData:long:subObject[1]": "banking"
}

now the problem being when you want to try perform edits on these fields with the [0] logstash and the ruby plugins interpret that as a sub object not as being part of the name. i have tried escaping with [ or [ and it still has the same effect

ie from a ruby script this does not work

event.remove('additionalData:long:subObject[1]')

i found some people with similar issues and they did a regex replace before parsing to json buy my issue is this being from kafka its already in a json input with " codec => json"

Answer 1

You can use an OpenSearch Ingest Pipeline with a regular expression to strip these brackets from your JSON document before it's indexed.

The mutate string processor would enable you to do this with ease.