I have a process that is writing a JSON data object from Kafka and putting some of the fields from this object via the elastic API into elastic search index.
I have to write two separate messages - one for the data object and another for the Elastic search index which is a subset of the data object.
My question is this: Can I augment the JSON metamodel so that I publish only one record format from Kafka, which contains both the full data object and the indexable fields, but only the indexable fields are loaded into Elastic search? Then I don't have to maintain two separate processes and keep them both synced up, I just have one process and JSON record.
I am not batch loading, so I cannot use the Bulk load API and 'index' field marker that this tool uses like this JSON bulk load API example