Unable to map otlp resource attributes into loki labels

Question

I'm using a spring boot application which sends telemetry data including logs to opentelemetry collector using opentelemetry agent.

The logs are successfully transferred to opentelemetry collector and received at the grafana loki but the Loki labels are not updated with OTLP resource and log attributes.

current config -> Loki Label -> host_name: host.name (as mentioned in otlp resource label) expected : Loki Label -> host_name: 06605506424f or any other attribute name eg : service.name

opentelemetry collector config file :

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:  
  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: host.name
  
  resource:
    attributes:
    - action: insert
      key: loki.attribute.labels
      value: service.name
    - action: insert
      key: loki.resource.labels
      value: host.name, container.id  
      
  batch:

exporters:
  jaeger:
    endpoint: "jaeger:14250"
    tls:
      insecure: true
  logging:
    loglevel: debug    
   
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"
    
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "default"    

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [health_check,pprof,zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [resource,attributes,batch]
      exporters: [logging,loki] 

opentelemetry collector logs :

Resource SchemaURL: https://opentelemetry.io/schemas/1.12.0
Resource attributes:
     -> container.id: STRING(06605506424f3396e1c92aeedc22337288177bc25c97af21beeda42a898e9be4)
     -> host.arch: STRING(amd64)
     -> host.name: STRING(06605506424f)
     -> os.description: STRING(Linux 5.10.16.3-microsoft-standard-WSL2)
     -> os.type: STRING(linux)
     -> process.command_line: STRING(/usr/lib/jvm/java-1.8-openjdk/jre:bin:java -javaagent:/dir/opentelagent.jar)
     -> process.executable.path: STRING(/usr/lib/jvm/java-1.8-openjdk/jre:bin:java)
     -> process.pid: INT(1)
     -> process.runtime.description: STRING(IcedTea OpenJDK 64-Bit Server VM 25.212-b04)
     -> process.runtime.name: STRING(OpenJDK Runtime Environment)
     -> process.runtime.version: STRING(1.8.0_212-b04)
     -> service.name: STRING(spring-api-gateway)
     -> telemetry.auto.version: STRING(1.18.0)
     -> telemetry.sdk.language: STRING(java)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(1.18.0)
     -> loki.attribute.labels: STRING(service.name)
     -> loki.resource.labels: STRING(host.name, container.id)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope com.apigateway.SpringApiGatewayApplication 
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2022-10-04 10:02:10.386 +0000 UTC
SeverityText: INFO
SeverityNumber: SEVERITY_NUMBER_INFO(9)
Body: Started SpringApiGatewayApplication in 10.038 seconds (JVM running for 16.182)
Attributes:
     -> loki.attribute.labels: STRING(host.name)
Trace ID: 
Span ID: 
Flags: 0
        {"kind": "exporter", "data_type": "logs", "name": "logging"}

grafana Loki

:

Answer 1

I think this is a known bug in the Loki exporter for OpenTelemetry. The workaround is to define the resource labels in the logRecord attribute section until the issue is fixed

Answer 2

This was initially answered by @Lewis, but it actually needs more details in order to solve the original problem.

The bug in the otel-collector when dealing with "loki.resource.labels" was fixed as of 0.67.0. But even after updating to a newer version, you may find that Loki does not accept labels containing a "." character. The following workaround sorted the problem for me:

resource: 
  attributes:  
    - action: insert 
      key: service_name 
      from_attribute: service.name 
    - action: insert 
      key: loki.resource.labels 
      value: service_name 

This first copies the "service.name" resource key to a new resource called "service_name". Then we tell Loki exporter about "service_name", which it will promote to a label and remove from the resources.