How to configure Grafana Agent to only skip too old logs?

Question

I try to use Grafana Agent (version 0.35) instead of Promtail to send logs to Grafana Loki (version 2.8.4). I want to send Docker container's logs, so I configured docker_sd_configs. But after starting Grafana Agent I couldn't find any logs with Grafana. But I can see an error message in Grafana Agent's logs.

Aug 25 10:02:15 test grafana-agent[12795]: ts=2023-08-25T10:02:15.806026191Z caller=client.go:430 level=error component=logs logs_config=default component=client host=172.31.11.242:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): entry for stream '{container=\"keycloak\", host=\"test\", source=\"stdout\"}' has timestamp too old: 2023-08-16T16:20:32Z, oldest acceptable timestamp is:2023-08-18T10:02:15Z"

Configuration

logs:
  configs:
  - name: default

    positions:
      filename: /tmp/positions.yaml

    clients:
      - url: http://172.31.11.242:3100/loki/api/v1/push
        external_labels:
          host: test

    scrape_configs:
    - job_name: docker
      docker_sd_configs:
          - host: unix:///var/run/docker.sock
            refresh_interval: 5s
      relabel_configs:
          - source_labels: ['__meta_docker_container_name']
            regex: '/(.*)'
            target_label: 'container'
          - source_labels: [__meta_docker_container_log_stream]
            target_label: source

Research

• I didn't have the problem with pure Promtail version 2.8.4. At least I didn't notice it.

• I read Loki/Promtail : parsing timestamp that are too old, but I don't want to change Loki's configuration to allow old logs. I just want to skip all too old logs.

• After restarting the Docker containers, Grafana Agent is reading all new logs. But I don't want to restart every container after installing Grafana Agent.

• I read journal reading: Timestamp too old errors, but this fix is only for the journal configuration.

• I read Timestamp error, but it is not about Docker container logs. Also I didn't find any official documentation about eventTime.

Question

How to configure Grafana Agent to only skip too old logs?

Answer 1

With the agent in static mode, you would have to find the equivalent in promtail config, which I do not know (see agent logs_config).

There is a new way to configure the Grafana Agent, the Flow mode. Using the Agent in Flow mode you can configure a loki.process with a drop stage block, here in the context of collecting kubernetes pods logs:

discovery.kubernetes "pods" {
  role = "pod"
}
loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.process.drop_old.receiver]
}

loki.process "drop_old" {
  forward_to = [loki.write.loki.receiver]
  stage.drop {
    older_than          = "1h"
    drop_counter_reason = "too old"
  }
}

loki.write "loki" {
  endpoint {
    url = "https://logs.grafana.net/loki/api/v1/push"
    basic_auth {
      username = ""
      password = ""
    }
  }
}

Hope that helps.