How can I route ECS log to S3 with using AWS Firelens

634 Views Asked by At

I am making ECS Service withe Terraform. Now I have to route my ECS Service log to S3 my bucket.

I looked it up and they told me to use AWS Firelens. But It dosen't work. Here is my setting.

resource "aws_ecs_task_definition" "xxx-terraform-task-definition" {
  family                   = "xxx-terraform-task"
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = var.fargate_cpu
  memory                   = var.fargate_memory
  container_definitions    = jsonencode([
    {
      name : var.app_name,
      image : var.app_image,
      cpu : var.fargate_cpu,
      memory : var.fargate_memory
      networkMode : "awsvpc",
      logConfiguration : {
        "logDriver" : "awslogs",
        "options" : {
          "awslogs-group" : var.xxx_logs,
          "awslogs-region" : var.aws_region,
          "awslogs-stream-prefix" : "ecs"
        }
      },
      portMappings : [
        {
          "containerPort" : 5001,
          "hostPort" : 5001
        }
      ]
    },
    {
      essential : true,
      image : "public.ecr.aws/aws-observability/aws-for-fluent-bit:latest",
      name : "log_router",
      firelensConfiguration : {
        "type" : "fluentbit",
        "config-file-type" : "file",
        "config-file-value" : "./firelens.config"
      },
      logConfiguration : {
        "logDriver" : "awslogs",
        "options" : {
          "awslogs-group" : "firelens-container",
          "awslogs-region" : var.aws_region,
          "awslogs-create-group" : "true",
          "awslogs-stream-prefix" : "firelens"

        }
      },
      memoryReservation : 50
    },
    {
      essential : true,
      image : "httpd",
      name : "app",
      logConfiguration : {
        "logDriver" : "awsfirelens",
        "options" : {
          "Name" : "s3",
          "region" : var.aws_region,
          "bucket" : "xxx-logs-bucket",
          "total_file_size" : "1M",
          "upload_timeout" : "1m",
          "use_put_object" : "On",
          "retry_limit" : "2"
        }
      },
      memoryReservation : 100
    }
  ])
}

This is my firelens.config

[OUTPUT]
    Name                         s3
    Match                        *
    bucket                       xxx-logs-bucket
    region                       ap-northeast-2
    total_file_size              50M
    use_put_object               on
    compression                  gzip
    s3_key_format                /$TAG/%Y/%m/%d/%H_%M_%S/$UUID.gz
    s3_key_format_tag_delimiters .-

After completing the terraform apply, when I check the ECS Service log, some logs are repeatedly generated over and over. It is never stop.

[error] [output:s3:s3.1] Could not send chunk with tag app-firelens-dd6592001de147d09588e566aa5e60ba
[error] [output:s3:s3.1] PutObject request failed
[error] [aws_client] could not sign request
[error] [signv4] Provider returned no credentials, service=s3
[ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.
[ warn] [imds] unable to evaluate IMDS version
[error] [aws_client] connection initialization error
[error] [aws_credentials] Failed to retrieve credentials for AWS Profile default
[error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist
[error] [aws_client] could not sign request
[error] [signv4] Provider returned no credentials, service=s3
[ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.
[ warn] [imds] unable to evaluate IMDS version
[error] [aws_client] connection initialization error
[error] [aws_credentials] Failed to retrieve credentials for AWS Profile default
[error] [aws_credentials] Shared credentials file /root/.aws/credentials does not exist

I would appreciate it if you could tell me which one I should handle.

0

There are 0 best solutions below