Can't Retrieve Logs by Tenant ID: Lambda Promtail Logs Loki/Grafana

259 Views Asked by At

I am attempting to setup Lambda Promtail to send cloudtrail logs to Loki (hosted in EKS). It appears from Lambda metrics that everything appears to be correct and I am not getting any errors on the client (lambda) side. I am using the recommended Grafana Terraform setup and am using the S3 add object event method.

module "cloudtrail-loki-promtail" {
  source = "../modules/loki-lambda-promtail"

  name = "cloud_trail_lambda_promtail"
  lambda_promtail_image = var.lambda_promtail_image
  lambda_vpc_subnets = []

  # Loki Server Config/Credentials
  write_address = var.loki_endpoint
  username = var.loki_username
  password = var.loki_password
  tenant_id = var.tenant_id
  batch_size = var.batch_size

  # For log aggregation through S3 (list)
  bucket_names = ["aws-cloudtrail-logs-<account-id>-<random>"]

  # For log aggregation through log group names (list)
  log_group_names = []

}

The promtail image is the latest one provided by Grafana (with no modifications). Basically following this setup --> https://grafana.com/docs/loki/latest/clients/lambda-promtail/

When I attempt to get the logs pulled into Grafana by tenant_id. I get the following error: Data source connected, but no labels received. Verify that Loki and Promtail is configured properly.

I am using basic_auth authentication and the X-Scope-OrgID to pull based on tenant_id.

Wondering if anyone has run into this issue and if they have a fix. Can't seem to find any helpful errors or documentation so far.

0

There are 0 best solutions below