Filtering Logs from Fluentd Docker Container

63 Views Asked by At

I am currently trying to filter the output of logs for my docker containers when I run (logs -f). I need to filter the logs since the fluentd container is especially noisy with many errors like:

cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/output.rb:897:in `emit_buffered'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/event_router.rb:115:in `emit_stream'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/event_router.rb:106:in `emit'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:355:in `on_message'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:226:in `block in handle_connection'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:263:in `block (3 levels) in read_messages'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:262:in `feed_each'
cluster-fluentd-1        |   2023-10-25 16:32:18 +0000 [warn]: #0 /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.16.2/lib/fluent/plugin/in_forward.rb:262:in `block (2 levels) in read_messages'

fluentd.conf file:

<system>
  log_level debug
  <source>
    @type  forward
    @id    input1
    @label @mainstream
    port  24224
  </source>

  <filter **>
    @type stdout
  </filter>

  <label @mainstream>
    <match docker.**>
      @type file
      @id   output_docker1
      path         /fluentd/log/docker.*.log
      symlink_path /fluentd/log/docker.log
      append       true
      timekey %Y%m%d
      timekey_wait   1m
      time_format       %Y%m%dT%H%M%S%z
    </match>
    <match **>44
      @type file
      @id   output1
      path         /fluentd/log/data.*.log
      symlink_path /fluentd/log/data.log
      append       true
      timekey %Y%m%d
      timekey_wait  10m
      time_format       %Y%m%dT%H%M%S%z
    </match>
  </label>
</system>

Adding the log level to the system configuration does remove these warnings and errors from the output, but then the rest of my fluentd.conf file is ignored as logs are no longer saved in the files as usual. I tried resolving the fluentd container errors but I have not been able to despite having the most updated version of fluentd and ruby gem installed. If anyone has any suggestions that allows for the fluentd container error logs to be ignored while also allowing log file persistence to remain I would appreciate it! TIA

0

There are 0 best solutions below