Fluent-bit grep filter is not excluding to tail input

133 Views Asked by At

I'm working with fluent-bit on my ARM64v8 SBC. It's doing a good job so far but one thing does not work.

In logs.conf I gather logs from a file and want to save them in Azure. This is working so far as expected.

[INPUT]
    Name        tail
    Tag         log.mender
    Refresh_Interval 10
    Path        /mnt/logs/mender/last

[FILTER]
    Name                     grep
    Alias                    ignore-common-logs
    Match                    log.*
    # Usual Mender client logs looping
    Exclude                  log /.*update-check [Sync] -> check-wait [Idle]*./
    Exclude                  log /.*check-wait [Idle] -> update-check [Sync]*./
    Exclude                  log /.*check-wait [Idle] -> inventory-update [Sync]*./
    Exclude                  log /.*request POST to <url>/api/devices/v2/depxt returned HTTP 404*./

[OUTPUT]
    Name        azure
    Match       log.mender
    Customer_ID <workspace_id>
    Shared_Key  <shared_key>
    Log_Type    Mender

I tried many things but the exclude in the grep is not working at all. The logs that I try to exclude are looping and spamming.

Sample from log file.

time="2023-10-19T13:31:39+02:00" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
time="2023-10-19T13:32:09+02:00" level=info msg="request POST to <url>/api/devices/v2/deployments/device/deployments/next returned HTTP 404"
time="2023-10-19T13:32:09+02:00" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
time="2023-10-19T13:32:09+02:00" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
time="2023-10-19T13:32:09+02:00" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
time="2023-10-19T13:32:09+02:00" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
time="2023-10-19T13:32:39+02:00" level=info msg="request POST to <url>/api/devices/v2/deployments/device/deployments/next returned HTTP 404"
time="2023-10-19T13:32:39+02:00" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
time="2023-10-19T13:32:39+02:00" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
time="2023-10-19T13:32:39+02:00" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
time="2023-10-19T13:32:39+02:00" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
time="2023-10-19T13:33:09+02:00" level=info msg="request POST to <url>/api/devices/v2/deployments/device/deployments/next returned HTTP 404"
time="2023-10-19T13:33:09+02:00" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]"
time="2023-10-19T13:33:09+02:00" level=info msg="State transition: check-wait [Idle] -> inventory-update [Sync]"
time="2023-10-19T13:33:10+02:00" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
time="2023-10-19T13:33:10+02:00" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"

Any suggestions what it might be? Is the order of definition important for fluent-bit? Fluent-bit is not producing any errors.

This is my fluent-bit.conf

@INCLUDE logs.conf
@INCLUDE system-monitor.conf

[SERVICE]
    Log_File      /var/log/fluentbit.log
    Daemon        on
    Parsers_File  parsers.conf
    HTTP_Server   Off
    storage.path              /mnt
    storage.sync              full
    storage.max_chunks_up     256
    storage.backlog.mem_limit 20M

[FILTER]
    Name    modify
    Match   *
    Add     serial ${SERIAL}
    Add     sku ${SKU}

[OUTPUT]
    name  stdout
    match *

Thanks in advance!

0

There are 0 best solutions below