Fluentbit and Fluentd in EFK Stack, why i need to use fluentd?

Question

Hey folks can anyone explain to me why does i need to use fluentd for aggregator? Below is the infrastructure i have been working on

Fluentbit(Log Forwarder) --> Fluentd(Data Processing and Aggregator) --> Elasticsearch --> Kibana

Sorry for not show picture, but you can find the topologhy what i mean in here : https://medium.com/redbox-techblog/building-an-open-data-platform-logging-with-fluentd-and-elasticsearch-4582de868398

Okay maybe this is a weird a question, but i still dont understand why i need it? when i just can use the fluentbit to forward the log straight to elasticsearch, why we must need the aggregator?

For more information i will be using a lot of fluentbit for couple of node(around 50 nodes i think)

I appriciate if you help me, and sorry for my bad english

Thank you

Answer 1

You don't need it, just make sure your Elasticsearch cluster is properly setup so it can handle the load with several incoming connections and traffic.

Answer 2

A log collector like fluentd or logstash makes sure that there is no risk of "bad data", which may cause problems while in production. Log collectors provides a unified logging layer, decoupling data sources from back-end systems.

Hope this helps.