I have a fluentbit running that scrapes json logs from a dir:
[PARSER]
Name json
Format json
Time_Key time
# Time_Format %llu
Time_Keep On
[PARSER]
Name extract
Format regex
Skip_Empty_Values On
Regex ^(?<event>[^,]*),(?<app>[^,]*),(?<user>[^,]*)$
{"level":33,"time":1710879528,"msg":"app_opened,"test_dashboard",user1"}
Which works fine. The logs are now send upstream to get stored. However, it can happen that a faulty configured log message gets saved in the same dir, too. Fluent-bit, unfortunately, does not discard un-parsable logs but tries to dump those as well.
Coming to my question: How can I select only those logs msg that got correctly parsed?
Config.yaml:
service:
flush: ${flush_interval}
grace: ${flush_interval}
log_level: info
http_server: On
http_listen: 0.0.0.0
http_port: 2020
pipeline:
inputs:
- name: tail
Read_from_Head: True
Mem_Buf_Limit: 2mb
parser: json
tag: userlogs
path: ${input_path}
filters:
- name: parser
match: '*'
parser: extract
Key_Name: msg
Reserve_Data: True
Preserve_Key: False
outputs:
# - name: file
# match: login
# path: /var/log/userlogs_output
- name: http
match: '*'
host: ${rec_host}
port: ${rec_port}
uri: ${rec_endpoint}
Retry_Limit: False
format: json
header_tag: FLUENT-TAG
log_response_payload: False