[Open Telemetry][otel-collector][prometheus] There's a huge delay between the sending of metrics and their processing

Question

I'm currently mounting an Open Telemetry - Prometheus stack to be able to process metrics, as a POC for now.

I'm sending continuous data to the Otel-collector, which then export it as Prometheus endpoint, which in turns gets scraped by a Prometheus server.

The problem is that it takes forever for a new metric to appear in Prometheus. I spent the evening yelling at my computer yesterday, only to find out that the metric appeared this morning by magic. I'm trying the same installation on a different server and I have the same result, no metrics yet after an hour.

What am I doing wrong ? I tried enabling debug logging on otel-collector, I keep seeing that the metric get "accumulated" by the exporter, but it's not available to be scraped.

Here's my configuration:

docker-compose.yml

version: '3.4'

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - 1888:1888 # pprof extension
      - 8888:8888 # Prometheus metrics exposed by the collector
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver
      - 4318:4318 # OTLP http receiver
      - 55679:55679 # zpages extension
    networks:
      - classroom-management_webnet

  prometheus:
    image: prom/prometheus
    ports:
      - 9090:9090
    networks:
      - classroom-management_webnet
    volumes: 
      - ./prometheus:/etc/prometheus
    command: --config.file=/etc/prometheus/prometheus.yml --log.level=debug

  grafana:
    image: grafana/grafana
    ports:
      - 3131:3131
    environment: 
      - GF_SERVER_HTTP_PORT=3131
    networks:
      - classroom-management_webnet

networks:
  classroom-management_webnet:
    external: true

otel-collector-config.yml

receivers:
  otlp:
    protocols:
      http:
      grpc:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [logging,prometheus]
  telemetry:
    logs:
      level: "debug"

prometheus.yml

scrape_configs:
- job_name: 'otel-collector'
  scrape_interval: 1s
  static_configs:
  - targets: ['otel-collector:8888']

I tried changing the otel-collector processor from batch to none to memory_limiter, with no success.

Answer 1

From your prometheus configuration, you are scraping the otel-collector's 8888 port which is where the collector emits its own metrics. Because you are using the prometheus exporter from the collector, you'll want to also scrape the prometheus exporter port (8889).