I am attempting to setup Lambda Promtail to send cloudtrail logs to Loki (hosted in EKS). It appears from Lambda metrics that everything appears to be correct and I am not getting any errors on the client (lambda) side. I am using the recommended Grafana Terraform setup and am using the S3 add object event method.
module "cloudtrail-loki-promtail" {
source = "../modules/loki-lambda-promtail"
name = "cloud_trail_lambda_promtail"
lambda_promtail_image = var.lambda_promtail_image
lambda_vpc_subnets = []
# Loki Server Config/Credentials
write_address = var.loki_endpoint
username = var.loki_username
password = var.loki_password
tenant_id = var.tenant_id
batch_size = var.batch_size
# For log aggregation through S3 (list)
bucket_names = ["aws-cloudtrail-logs-<account-id>-<random>"]
# For log aggregation through log group names (list)
log_group_names = []
}
The promtail image is the latest one provided by Grafana (with no modifications). Basically following this setup --> https://grafana.com/docs/loki/latest/clients/lambda-promtail/
When I attempt to get the logs pulled into Grafana by tenant_id. I get the following error: Data source connected, but no labels received. Verify that Loki and Promtail is configured properly.
I am using basic_auth authentication and the X-Scope-OrgID to pull based on tenant_id.
Wondering if anyone has run into this issue and if they have a fix. Can't seem to find any helpful errors or documentation so far.