Optimization: Improve Loki logs performance while fetching from grafana

59 Views Asked by At

I have a single monolithic loki instance installed from the loki-stack chart. This is running on a kubernetes cluster, taking logs from 3 other clusters too. The log volume is pretty high and needed. I have s3 configured to store these logs.

However, when I try to fetch these logs in grafana ui from explore, and when I click on label in query builder, it takes forever to load the labels. Is it because of something am I doing wrong? Some queries are evening timing out and trying to consuming CPU upto 1500m.

here is my loki config:

loki:
  enabled: true
  isDefault: true
  url: http://{{(include "loki.serviceName" .)}}:{{ .Values.loki.service.port }}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  datasource:
    jsonData: "{}"
    uid: ""
  config:
      schema_config:
        configs:
          - from: 2021-05-12
            store: boltdb-shipper
            object_store: s3
            schema: v11
            index:
              prefix: loki_index_
              period: 24h
      storage_config:
        aws:
          s3: <s3 buckets>
          s3forcepathstyle: true
          bucketnames: <s3 bucket name>
          region: <region>
          insecure: false
          sse_encryption: false
        boltdb_shipper:
          shared_store: s3
          cache_ttl: 24h
      limits_config:
        ingestion_rate_mb: 24
        ingestion_burst_size_mb: 24
        per_stream_rate_limit: "6MB"
        per_stream_rate_limit_burst: "15MB"
  1. Should I consider Loki-distributed if I query requirements and log ingestions are higher?
  2. Will it help if I add more replicas of current monolithic setup?
0

There are 0 best solutions below