Fluentbit Syslog Output

Question

I am attempting to output a particular field of alermanager alerts sent to fluentbit rather than to a syslog server.

Now I'm having difficulty capturing the required field I need because it is nested within the JSON alert that is being sent.

alermanager alert example:

{
  "receiver": "fluentbit-webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "KubeJobFailed",
        "condition": "true",
        "container": "kube-state-metrics",
        "endpoint": "http",
        "instance": "10.42.6.188:8080",
        "job": "kube-state-metrics",
        "job_name": "helm-install-aws-ebs-csi-driver",
        "namespace": "kube-system",
        "pod": "prometheus-operator-kube-state-metrics-59c8dc555f-l7dlv",
        "prometheus": "monitoring/prometheus-operator-kube-p-prometheus",
        "service": "prometheus-operator-kube-state-metrics",
        "severity": "warning"
      },
      "annotations": {
        "description": "Job kube-system/helm-install-aws-ebs-csi-driver failed to complete. Removing failed job after investigation should clear this alert.",
        "runbook_url": "https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubejobfailed",
        "summary": "Job failed to complete."
      },
      "startsAt": "2023-10-05T09:21:25.327Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus.monitoring.core.oxygen.example.com/graph?g0.expr=kube_job_failed%7Bjob%3D%22kube-state-metrics%22%2Cnamespace%3D~%22.%2A%22%7D+%3E+0&g0.tab=1",
      "fingerprint": "1a5cd56a32bc18c2"
    }
  ],
  "groupLabels": {
    "namespace": "kube-system"
  },
  "commonLabels": {
    "alertname": "KubeJobFailed",
    "condition": "true",
    "container": "kube-state-metrics",
    "endpoint": "http",
    "instance": "10.42.6.188:8080",
    "job": "kube-state-metrics",
    "job_name": "helm-install-aws-ebs-csi-driver",
    "namespace": "kube-system",
    "pod": "prometheus-operator-kube-state-metrics-59c8dc555f-l7dlv",
    "prometheus": "monitoring/prometheus-operator-kube-p-prometheus",
    "service": "prometheus-operator-kube-state-metrics",
    "severity": "warning"
  },
  "commonAnnotations": {
    "description": "Job kube-system/helm-install-aws-ebs-csi-driver failed to complete. Removing failed job after investigation should clear this alert.",
    "runbook_url": "https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubejobfailed",
    "summary": "Job failed to complete."
  },
  "externalURL": "http://alertmanager.monitoring.core.oxygen.example.com",
  "version": "4",
  "groupKey": "{}/{severity=\"warning\"}:{namespace=\"kube-system\"}",
  "truncatedAlerts": 0
}

How do I retrieve the "description" value that is nested within the "commonAnnotations" key?

here is an example of another fluentbit syslog output I am using for a non-nested json log

    [OUTPUT]
        Name syslog
        Match syslog.*
        Host bastion.dev.oxyproj.net
        Port 514
        Retry_Limit false
        Mode tcp
        Syslog_Format rfc5424
        Syslog_MaxSize 65536
        Syslog_Hostname_Key hostname
        Syslog_Appname_Key appname
        Syslog_Procid_Key procid
        Syslog_Msgid_Key msgid
        Syslog_SD_Key uls@0
        Syslog_Message_Key msg

this syslog output example captures the "msg" value in a non-nested json log.

Thank you.

Answer 1

Use the nest filter to extract the nested key.

Tested on fluent-bit 2.1.10 and syslog-ng 3.36.1.

fluent-bit config:

[INPUT]
    name dummy
    dummy {"commonAnnotations": {"description": "some-description", "runbook_url": "some-url"}}

[FILTER]
    name nest
    match *

    operation lift
    nested_under commonAnnotations

[OUTPUT]
    name syslog
    match *

    host 127.0.0.1
    port 514
    mode tcp

    syslog_format rfc5424
    syslog_message_key description

[OUTPUT]
    name stdout
    match *

syslog-ng config:

source dummy {
  network(
    ip("127.0.0.1")
    port(514)
    transport("tcp")
    flags(syslog-protocol)
  );
};

destination file {
  file("/var/log/dummy.log");
};

log {
  source(dummy);
  destination(file);
};

fluent-bit output:

Jan 13 14:23:04 fluent-bit-exporter fluent-bit[23092]: [0] dummy.0: [[1705144983.799075055, {}], {"description"=>"some-description", "runbook_url"=>"some-url"}]

syslog-ng output:

2024-01-13T14:23:04+03:00 127.0.0.1 some-description