I am using Firelens with Fargate to forward logs to an OpenSearch cluster (in the same VPC) as the Fargate task. Here's my configuration:
"logDriver": "awsfirelens",
"options": {
"Aws_Auth": "On",
"Aws_Region": "us-east-1",
"Buffer_Size": "False",
"HTTP_Passwd": "MY_OPEN_SEARCH_PASSWORD",
"HTTP_User": "masteruser",
"Host": "vpc-fake-domain-dmgfndgjdfgdfe.us-east-1.es.amazonaws.com",
"Index": "logstash-*",
"Logstash_Format": "On",
"Match": "*",
"Name": "opensearch",
"Port": "443",
"Replace_Dots": "On",
"Suppress_Type_Name": "On",
"Trace_Error": "On",
"Trace_Output": "On",
"Type": "_doc",
"tls": "On",
"tls.verify": "Off"
},
"secretOptions": []
}
Even though I've Trace_Error set to On, I am not getting any details on the issue other than:
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
</body>
</html>
This appears to be a generic 400 HTTP error page that Fluent bit appears to be receiving. Is there a way for me to find out what exactly is causing this?
Here's the trace output for the request:
{"create":{"_index":"logstash-2023.09.15"}}
{"@timestamp":"2023-09-15T22:17:40.548Z","source":"stdout","log":"10.0.44.164 - - [15/Sep/2023:22:17:40 +0000] Log Message","container_id":"5a23495fc387471e858b8b0be1b3d57a-2521534270","container_name":"bundle-web","ecs_cluster":"fargate-cluster","ecs_task_arn":"arn:aws:ecs:us-east-1:XXXXXXXX:task/fargate-cluster/XXXXXX","ecs_task_definition":"dev-bundle-web:37"}