How to scrape stopped Docker containers?

Question

I use Grafana Agent to scrape all my Docker containers. Therefore, I use Prometheus' <docker_sd_config>. It works for running Docker containers, but not for stopped Docker containers. No alerts are sent.

Configuration

  configs:
    - name: agent
      scrape_configs:

      - job_name: docker
        metrics_path: '/prometheus'
        docker_sd_configs:
          - host: unix:///var/run/docker.sock
        relabel_configs:
          - source_labels: [__meta_docker_port_private]
            regex: '8081'
            action: keep
          - source_labels: ['__meta_docker_container_name']
            regex: '/(.*)'
            target_label: 'job'

Alert rule

- alert: ServerDown
    expr: up{} < 1
    for: 15m
    annotations:
      summary: Server down

Docker containers

f8070378f38b   gitlab.mycomp.com:5005/test/my-app1:latest    "java -jar app.jar"      2 weeks ago     Up 12 hours (healthy)        0.0.0.0:8205->8081/tcp, :::8205->8081/tcp                                                                                                                 my-app1
c4a5042b9fcf   gitlab.mycomp.com:5005/test/my-app2:latest    "java -jar app.jar"      2 weeks ago     Exited (143) 40 seconds ago                                                                                                                                                             my-app2

Research

• If I use <static_config> it works, but I don't want to list all my Docker containers manually. I want to use the service discovery.

Question

How to scrape stopped Docker containers with Grafana Agent?

Answer 1

Sadly there seem to be no easy switch to enable that. The API used to retrieve the list of containers (https://docs.docker.com/engine/api/v1.40/#tag/Container) has the option to show all but I don't see an option to use it in docker_sd_configs. However, there is an option to specify filters. I'm not sure if it would work, but maybe if you can create such a filter, that'll cover all containers. Something like this:

docker_sd_configs:
- host: unix:///var/run/docker.sock
  # Optional filters to limit the discovery process to a subset of available
  # resources.
  # The available filters are listed in the upstream documentation:
  # https://docs.docker.com/engine/api/v1.40/#operation/ContainerList
  filters:
  # Not sure which is correct, try either this:
  - name: status
    values: created, restarting, running, removing, paused, exited, dead
  # or this:
  - name: status
    values: created
  - name: status
    values: restarting
  - name: status
    values: running
  - name: status
    values: removing
  - name: status
    values: paused
  - name: status
    values: exited
  - name: status
    value: dead