I'm trying to use the elastic search filter on logstash for make some Data Enrichment.
I got two indexes, and my goal it's get some data from one of them and add it to the other.
I configured a logstash filter who search in my elasticsearch and if there is a match the output goes to the index.
But my filter it's not working propery because when I test the filter i got this error
[WARN ] 2020-10-02 19:23:09.536 [[main]>worker2] elasticsearch - Failed to query elasticsearch for previous event {:index=>"logstash-*", :error=>"Unexpected character ('%' (code 37)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n
I think there it's some issue between the variable in the template and the elastic search
my logstash it's a 7.3.2 and my ES an 7.4.2
here it's my settings
Logstash.conf
input {
http{ }
}
filter {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logstash-*"
query_template => "search-by-ip.json"
fields => {
"id" => "[suscriberid]"
}
}
}
output {
stdout { codec => rubydebug }
}
-----------------
search-by-ip.json
{
"size": 1,
"query": { "match":{"IP": %{[ip]} } }
}
-------------------
testcase.sh
curl -XPOST "localhost:8080" -H "Content-Type: application/json" -d '{
"size": 1,
"query": { "match":{"ip": "192.168.1.4" }}
}'
```
Thanks!
If you ever process an event that does not have an [ip] field then the sprintf reference will not be substituted and you will get that error.
Note that ip and IP are different fields. Not sure if the %{[ip]} requires double quotes around it.