I have a relatively low frequency log file (1 per second), I simply read from filebeat, push to external redis (net latenct ~5ms), on the same server as redis I pull via logstash and push to elasticsearch.
Nothing is really being overloaded CPU-wise but I still see a delay of ~4-10 seconds. How can I minimize that and make it nearly real-time?
Filebeat to Redis uses list with bulk_max_size: 0 and flush.timeout: 0s
Logstash uses 4 threads.