How to change prometheus jmx exporter rules in spark-operator image working as extension of gcr.io/spark-operator/spark:v3.0.0-hadoop3

Question

We are trying to substitute the prometheus.yaml in /etc/metrics/conf as it hosts older rules. We try to copy new rules file:

RUN mkdir -p /etc/metrics/conf
COPY conf/prometheus.yaml /etc/metrics/conf/prometheus.yaml

But the rules are not picked up and prometheus still ignores the new metrics.

Answer 1

Sad truth is that in fact the folder and file is alredy there (looks like a symlink to /etc/metrics/conf/..data/prometheus.yaml location).

We had to work it around with

1. copying files to new location

   COPY conf/prometheus.yaml /etc/metrics/conf2/prometheus.yaml

2. add configFile parameter to the helm chart:

   configFile: "/etc/metrics/conf2/prometheus.yaml"

That causes javaagent's properties passed to jmx-exporter to use this new prometheus.yaml location