Kiali is not working with VictoriaMetricsHello

Question

Was anyone was able to make kiali visualise mesh using VictoriaMetrics instead of Prometheus?

When I use prometheus and kiali setup from istio samples then Mesh Visualisation works.

But when I replace prometheus with Victoria Metrics (agent, select, insert, storage) then kiali simply showing empty graph.

I have checked that both Prometheus and Victoria Metrics have the same istio_requests_total metric.

But when I use Victoria Metrics select URL in spec.external_services.prometheus.url the graph comes empty.

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  namespace: istio-system
spec:
  auth:
    strategy: anonymous
  external_services:
    istio:
      config_map_name: istio-1-14
      url_service_version: http://istiod-1-14:15014/version
    prometheus:
      url: http://vmselect-example-vmcluster-persistent.poc.svc.cluster.local:8481/select/0/prometheus/

In logs I see two errors related to the fact that vm-select does not have corresponding endpoints

2022-07-15T19:25:13Z ERR Failed to fetch Prometheus configuration:     bad_response: readObjectStart: expect { or n, but found r, error found      in #1 byte of ...|remoteAddr:|..., bigger context ...|remoteAddr:      "10.4.34.83:57468"; requestURI: /select|... 
2022-07-15T19:25:13Z ERR Failed to fetch Prometheus flags: bad_response: readObjectStart:     expect { or n, but found r, error found in #1 byte of     ...|remoteAddr:|..., bigger context ...|remoteAddr:     "10.4.34.83:57468"; requestURI: /select|...

and multiple warnings

 2022-07-15T19:35:28Z WRN Skipping      {destination_canonical_revision="v1",     destination_canonical_service="microservice",      destination_cluster="Kubernetes",     destination_service="microservice.poc.svc.cluster.local",      destination_service_name="microservice",     destination_service_namespace="poc",     destination_workload="microservice",     destination_workload_namespace="poc", request_protocol="http",     response_code="200", response_flags="-", source_canonical_revision="latest",     source_canonical_service="istio-ingressgateway-internal",  source_cluster="Kubernetes"}, missing expected TS labels

Here is my VMPodScrape which I expect will take all metrics from all pods

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: all-scrape
  namespace: poc
spec:
  podMetricsEndpoints:
    - scheme: http
      path: /stats/prometheus
      targetPort: 15090
  selector: {}
  namespaceSelector:
    any: true

Answer 1

Error messages don't look critical, in case of error kiali should use default values. As far as I understand, it tries to recognize scrape interval and retention based on prometheus configuration file and flags.

I think you have an issue with relabeling config, it drops labels required for kiali.

There is docs with needed labels by metric name: https://kiali.io/docs/faq/general/#which-istio-metrics-and-attributes-are-required-by-kiali

I'd recommend check scrape config at VMAgent. Probably relabeling rules are outdated.

Answer 2

For me I did not use the VMPodScrape,

I used the simplest VictoriaMetrics Single in Kubernetes via Helm charts based on the official guide (https://docs.victoriametrics.com/guides/k8s-monitoring-via-vm-single.html) Note: I install install on "victoria-metric" namespace.

After the installation, I edit the configmap, which based on istio-1.17.1/samples/addons/prometheus.yaml , the config map is as below.

kind: ConfigMap
apiVersion: v1
metadata:
  annotations:
    meta.helm.sh/release-name: vmsingle
    meta.helm.sh/release-namespace: victoria-metric
  labels:
    app: server
    app.kubernetes.io/instance: vmsingle
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: victoria-metrics-single
    helm.sh/chart: victoria-metrics-single-0.8.55
  name: vmsingle-victoria-metrics-single-server-scrapeconfig
  namespace: victoria-metric
data:
  scrape.yml: |2

    global:
      scrape_interval: 15s

    scrape_configs:
    - job_name: victoriametrics
      static_configs:
      - targets:
        - localhost:8428
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-apiservers
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: default;kubernetes;https
        source_labels:
        - __meta_kubernetes_namespace
        - __meta_kubernetes_service_name
        - __meta_kubernetes_endpoint_port_name
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
    - honor_labels: true
      job_name: kubernetes-service-endpoints
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: drop
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: service
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node
    - honor_labels: true
      job_name: kubernetes-service-endpoints-slow
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_service_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_service_name
        target_label: service
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node
      scrape_interval: 5m
      scrape_timeout: 30s
    - honor_labels: true
      job_name: prometheus-pushgateway
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - action: keep
        regex: pushgateway
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
    - honor_labels: true
      job_name: kubernetes-services
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module:
        - http_2xx
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
      - source_labels:
        - __address__
        target_label: __param_target
      - replacement: blackbox
        target_label: __address__
      - source_labels:
        - __param_target
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - source_labels:
        - __meta_kubernetes_service_name
        target_label: service
    - honor_labels: true
      job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: drop
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: drop
        regex: Pending|Succeeded|Failed|Completed
        source_labels:
        - __meta_kubernetes_pod_phase
    - honor_labels: true
      job_name: kubernetes-pods-slow
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: drop
        regex: Pending|Succeeded|Failed|Completed
        source_labels:
        - __meta_kubernetes_pod_phase
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true

Here is my kiali configmap.

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    version: v1.63.1
  name: kiali
  namespace: istio-system
data:
  config.yaml: |
    ...
    external_services:
      custom_dashboards:
        enabled: true
      istio:
        root_namespace: istio-system
      prometheus:
        url: http://vmsingle-victoria-metrics-single-server.victoria-metric.svc.cluster.local:8428/

Everything is working in my setup.

During my debug journey to make it work, you may port-forwrd the traffic to victoria_metric pod to see the debug WebUI

kubectl port-forward <victoria_metric_pod_name> 8428:8428

then access VictoriaMetric on http://localhost:8428/targets This page will list the scape target in here, if you configure victoria metric scape correctly, we should see "kubernetes-pods" job up and running.

Here is the Kiali graph in my setup. Notice that in the Kiali pod, there is some error in the log ... but the graph is still showing up.