We are logging Restify requests in ElasticSearch using Logstash and Bunyan. However, when including the JSON body in logging the index gets conflicts because fields with the same name sometimes have different types.
One example is when req.body is sometimes a string and sometimes an object. We've worked around that by always setting body to an object (since our restify API is not supposed to receive strings for any valid requests).
However the problem keeps occurring for fields in the body object. We can't really control what a client sends in, and if a request contains a string where a number is expected the elasticsearch index has already typed that field to number.
Is there any way of continually fixing this apart from checking and potentially replacing every field posted in the body? Converting the body from an object to a string before logging it would work, but that would seriously reduce its usability in Kibana when making visualizations.
If a field needs to contain both a string or a number, you'll have to define it as a string. Otherwise, elasticsearch will just gladly drop the event when there's a mismatch.
Elastic is planning to add a "dead-letter" function to logstash, so you could keep the field as a number and have any events that came through as a string be directed there instead of elasticsearch.
To keep elasticsearch from making the field a number based on the first data received, you may want to disable dynamic mapping or setup a template to better help with the mapping.