Docker + Fluentd in K8s for log rotation: Does Docker need to know the existence of Fluentd?

Question

I am trying to understand the interaction between Docker and Fluentd in a K8s cluster. I have seen places where you need to configure Docker to output to a logging driver, and Fluentd can be used as logging driver, like here.

On the other hand, I have seen posts (like this or this) where Docker does not know the existence of Fluentd as a DaemonSet.

My whole intention is to do log rotation, however I am not sure if having Fluentd in place will actually rotate the logs Docker writes on, so I do not end up with the whole storage space in the node taken up by the logs over time. Is it enough to use FluentD DaemonSet without Docker knowing the existence of Fluentd?, o I need to somehow connect Docker to Fluentd with a driver as well?

Answer 1

Per official k8s logging architecture docker (or any other runtime) does not need to know about FluentBit, Fluentd, Filebeat, or any other log collector you use. In fact, you can use multiple log collectors a time!

The same document states that k8s is not responsible for log rotation, so you set up a logrotate yourself. Fluentd/FluentBit daemon on the other end also does not rotate log files, but it does able to track log rotation and adjust the tail cursor accordingly (by default).

By far the easiest way to implement the architecture is

• Leave kubelet & docker settings at default
• Ensure the app logs stdout/stderr
• Ensure there's logrotate: many k8s worker AMIs, e.g. EKS already have it.
• Setup FluentBit log collector daemonset https://github.com/helm/charts/tree/master/stable/fluent-bit