logstash with kafka input plugin logs are not getting ingesting

18 Views Asked by At

This is my current ELK setup. Beats => kafka => logstash => elasticsearch => KIbana.

Initially i used SQS but it is incurring lot of cost to us hence trying to use kafka as the replacement. I have created 2 node cluster kafka and created topics.

This is my logstash kafka config look like.

input {
  kafka {
    bootstrap_servers => "server1:9092,server2:9092"
    topics => ["topic1"]
    codec  => json
    consumer_threads => 100
}
}

There are multiple grok patterns. In the filtering. I can see logs are coming to kafka and to logstash and getting indexed but i don't see anything in kibana. Logs in logstash or elasticsearch doesn't give me any useful info to debug further.

0

There are 0 best solutions below