Is it possible to monitor a whole directory with fluentd?

Question

I would like to set up log forwarding as part of a deployment process. The activity of the machines will be different but they will all log to specific places (notably /var/log).

Is it possible to configure fluentd so that it monitors a whole directory? (including the ability to pick up files which pop-up while it is active)

I know that in_tail can do this for a given, specified file but the documentation does not mention a whole directory.

---

There is an ideal exact duplicate of this question from 2014 which points to the tail_ex plugin. Unfortunately its description mentions that

Deprecated: Fluentd has the features of this plugin since 0.10.45. So, the plugin no longer maintained

I still could not find the mentioned features.

Answer 1

Using the wildcard support within Fluentd's in_tail plugin this is absolutely possible. In the path section you would specify the /var/log/* directory and Fluentd will automatically skip files that are non-readable.

Additionally, if you write new files to this directory Fluentd will periodically scan based on the configuration item https://docs.fluentd.org/v0.12/articles/in_tail#refreshinterval

Some notes: If you use Treasure Data's packaged version of Fluentd, td-agent then you need to ensure that the files you want to tail are readable by the td-agent user that is provisioned as part of that install.

Lastly, if you need to securely read these files you may consider Treasure Data's Enterprise Fluentd offering