Spark Operator and jmx_exporter failing

Question

I've just migrated k8s to 1.22 and with this version spark-operator:1.2.3 didn't work. I've followed the info at the internet and upgraded to 1.3.3, however all my spark apps are failing with the same error: Caused by: java.io.FileNotFoundException: /etc/metrics/conf/prometheus.yaml (No such file or directory) at java.base/java.io.FileInputStream.open0(Native Method) at java.base/java.io.FileInputStream.open(FileInputStream.java:219) at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) at java.base/java.io.FileReader.<init>(FileReader.java:75) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:78) at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29) ... 6 more *** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422 FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed

It used to work on previous version.... unfortunately, I cannot downgrade k8s. Can you please assist? PS: there are no additional options passed to executor, just a path to jmx_exporter_0.15

Answer 1

I think your new application requires that prometheus be running in your cluster and it also expects to find the configuration file for prometheus at the path /etc/metrics/conf/prometheus.yaml. Such files are generally setup by creating a ConfigMap in your cluster and then mounting it to every pod that needs it.

My guess is, during the upgrade of spark a step was missed/not provided which was to install prometheus in your cluster before installing your spark applications which used that installation as a dependency. This is the case since you are trying to use a prometheus exporter and if a prometheus installation doesn't exist already, it will not work.

You can try going through the installation again and checking where prometheus comes into play and ensure that this configuration file is provided to your applications.