I am currently trying to filter the output of logs for my docker containers when I run (logs -f). I need to filter the logs since the fluentd container is especially noisy with many errors like:
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:897:in `emit_buffered'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/event_router.rb:115:in `emit_stream'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/event_router.rb:106:in `emit'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:355:in `on_message'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:226:in `block in handle_connection'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:263:in `block (3 levels) in read_messages'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:262:in `feed_each'
cluster-fluentd-1 | 2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:262:in `block (2 levels) in read_messages'
fluentd.conf file:
<system>
log_level debug
<source>
@type forward
@id input1
@label @mainstream
port 24224
</source>
<filter **>
@type stdout
</filter>
<label @mainstream>
<match docker.**>
@type file
@id output_docker1
path /fluentd/log/docker.*.log
symlink_path /fluentd/log/docker.log
append true
timekey %Y%m%d
timekey_wait 1m
time_format %Y%m%dT%H%M%S%z
</match>
<match **>44
@type file
@id output1
path /fluentd/log/data.*.log
symlink_path /fluentd/log/data.log
append true
timekey %Y%m%d
timekey_wait 10m
time_format %Y%m%dT%H%M%S%z
</match>
</label>
</system>
Adding the log level to the system configuration does remove these warnings and errors from the output, but then the rest of my fluentd.conf file is ignored as logs are no longer saved in the files as usual. I tried resolving the fluentd container errors but I have not been able to despite having the most updated version of fluentd and ruby gem installed. If anyone has any suggestions that allows for the fluentd container error logs to be ignored while also allowing log file persistence to remain I would appreciate it! TIA