How to disable JSON format and send only the log message to Cloudwatch with Fluentbit?

Question

I am trying to setup Firelens for my Fargate tasks. I would like to send logs to multiple locations, Cloudwatch and Elasticsearch.

But only to Cloudwatch I want to disable JSON format and send only the log message as it is.

I have the below configuration for Cloudwatch output.

[OUTPUT]
    Name cloudwatch
    Match *
    auto_create_group true
    log_group_name /aws/ecs/containerinsights/$(ecs_cluster)/application
    log_stream_name $(ecs_task_id)
    region eu-west-1

Currently logs are coming like this,

{
    "container_id": "1234567890",
    "container_name": "app",
    "log": "2021/08/10 18:42:49 [notice] 1#1: exit",
    "source": "stderr"
}

I want only the line,

2021/08/10 18:42:49 [notice] 1#1: exit

in Cloudwatch.

Answer 1

I had a similar issue using just CloudWatch where everything was wrapped in JSON - I imagine it'll be the same when using several targets.

The solution was to add the following to the output section:

   log_key log

This tells Fluent Bit to only include the data in the log key when sending to CloudWatch.

The docs have since been updated to include that line by default in this PR.