How to add custom labels in Promtail Config

Question

I need to Extract logs data and append as a new label, below is the sample log example:

Sample Log Message: 2022-12-21T11:48:00,001 [schedulerFactor_Worker-4, , ] INFO [,,] [userAgent=] [system=,component=,object=] [,] [] c.s.f.s.scheduler.SchedulerTask - sync process started on 2022-12-21T06:48:00.000780 for sync pair :17743b1b-a067-4478-a6d8-7b1cff04175a for JobId :dc8dc0dd-fdb9-4873-af55-1c70ba2047a5

New Labels needed in logs: sync_pair =17743b1b-a067-4478-a6d8-7b1cff04175a JobId =dc8dc0dd-fdb9-4873-af55-1c70ba2047a5

My promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  http_listen_address: 0.0.0.0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containers
      __path__: /var/lib/docker/containers/*/*log
- job_name: dockerlogs
  file_sd_configs:
  - files:
    - /etc/promtail/promtail-targets.yml
  relabel_configs:
  - source_labels: [job]
    target_label: job
  - source_labels: [__address__]
    target_label: container_id
  - source_labels: [container_id]
    target_label: __path__
    replacement: /var/lib/docker/containers/*/*log


  pipeline_stages:
  - match:
      selector: '{job="varlogs"}'
      stages:
      - regex:
          expression: '(?P<sync_pair>sync_pair)' '(?P<job_id>job_id)'
      - labels:
          sync_pair:
          job_id:

I have added pipeline stages, but it's not showing any labels. i.e. sync_pair and JobId, this labels should be shown in logs after query.

2022-12-21T11:48:00,001 [schedulerFactor_Worker-4, , ] INFO [,,] [userAgent=] [system=,component=,object=] [,] [] c.s.f.s.scheduler.SchedulerTask - sync process started on 2022-12-21T06:48:00.000780 for sync pair :17743b1b-a067-4478-a6d8-7b1cff04175a for JobId :dc8dc0dd-fdb9-4873-af55-1c70ba2047a5

This two must be shown in Log Labels: **sync_pair =17743b1b-a067-4478-a6d8-7b1cff04175a JobId =dc8dc0dd-fdb9-4873-af55-1c70ba2047a5 **

Check Image ---> https://i.stack.imgur.com/1h27v.png

I want sync_pair and JobId in my Labels.

Answer 1

promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  http_listen_address: 0.0.0.0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  pipeline_stages:
  - regex:
      # extracts only sync pair and JobId from the log line
      expression: '.*sync pair :(?P<syncpair>[a-zA-Z0-9_-]{30,36}).*JobId :(?P<jobid>[a-zA-Z0-9_-]{30,36})'
  - labels:
      # sources extracted syncpair as label 'sync pair' value and jobid as label 'JobId'
      syncpair:
      jobid:
- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containers
      __path__: /var/lib/docker/containers/*/*log
- job_name: dockerlogs
  file_sd_configs:
  - files:
    - /etc/promtail/promtail-targets.yml
  relabel_configs:
  - source_labels: [job]
    target_label: job
  - source_labels: [__address__]
    target_label: container_id
  - source_labels: [container_id]
    target_label: __path__
    replacement: /var/lib/docker/containers/*/*log

Try this tool -> https://regex101.com/r/rgp49r/1

Reference -> Using Promtail to sum log line values - Pipeline Stages - Metrics

Check output image for Labels