How can I relabel ECS cadvisor labels in prometheus config?

Question

I am trying to rename same labels from cadvisor in the prometheus config. The cadvisor names are just too long, so I have the following config for relabeling.

  - job_name: 'cadvisor job foo'
scrape_interval: 60s
ec2_sd_configs:
  - role_arn: 'arn:aws:iam::$id:role/foo'
    region: 'us-west-2'
    port: 8484
relabel_configs:
  - source_labels: [__meta_container_label_com_amazonaws_ecs_cluster]
    target_label: cluster_name
  - source_labels: [__meta_container_label_com_amazonaws_ecs_container_name]
    target_label: container_name  

Unfortunately that does not seem to work. Is this even possible to relabel cadvisor labels in the prometheus config?

Answer 1

Are you sure that the label __meta_container_label_com_amazonaws_ecs_cluster exists? I'm running Cadvisor in a AWS ECS cluster and I get the container name from container_label_com_amazonaws_ecs_container_name. It is not a meta label but extracted by Cadvisor from the Docker labels.

__meta.+ labels are retrieved by relabel_configs. Here is what that part of my config looks like:

  - job_name: cadvisor
    ec2_sd_configs:
      - region: eu-central-1
        port: 28080
        filters:
          - name: tag:aws:autoscaling:groupName 
            values:
              - data-dev
              - data-prod

    relabel_configs:
      - action: replace
        source_labels: [__meta_ec2_instance_id]
        target_label: instance_id
      - action: replace
        source_labels: [__meta_ec2_instance_state]
        target_label: instance_state
      - action: replace
        source_labels: [__meta_ec2_instance_type]
        target_label: instance_type

    metric_relabel_configs:

      # Drop series.
      - action: drop
        source_labels: [__name__]
        regex: "(container_tasks_state|container_cpu_load_average_10s)"
      
      # Drop all series that do not have a 'name' label.
      - action: keep
        source_labels: [name]
        regex: (.+)

      # Drop all series that do not have a '/ecs/' c group prefix and a proper 
      # container id after the third forword slash.
      - action: keep
        source_labels: [id]
        regex: ^/ecs/.+/.+

      # Drop label 'instance_type'.
      - action: labeldrop
        regex: instance_type

      # Rewrite 'id' label to only include short form container id.
      - action: replace
        source_labels: [id]
        target_label: id
        regex: ^/ecs/.+/(.{0,12}).*
        replacement: $1

      # Rename label task family.
      - action: replace
        source_labels: [container_label_com_amazonaws_ecs_task_definition_family]
        target_label: task_family
      - action: labeldrop
        regex: container_label_com_amazonaws_ecs_task_definition_family

      # Overwrite 'name' with value of container name label.
      - action: replace
        source_labels: [container_label_com_amazonaws_ecs_container_name]
        target_label: name
      - action: labeldrop
        regex: container_label_com_amazonaws_ecs_container_name

      # namespace
      - action: replace
        source_labels: [container_label_promstack_namespace]
        target_label: namespace
      - action: labeldrop
        regex: container_label_promstack_namespace

      # api_type
      - action: replace
        source_labels: [container_label_promstack_api_type]
        target_label: api_type
      - action: labeldrop
        regex: container_label_promstack_api_type