Scraping custom metrics into Prometheus from a Pyspark cluster running on K8s using the Python Client

Question

I am starting to work with PySpark structured streaming applications running on a Kubernetes cluster deployed using the K8s Spark Operator. I would like to send metrics to monitor these streams to prometheus.

I can successfully scrape the standard Spark metrics to Prometheus using a PodMonitor from the spark-ui (4040) port. I could not work out how to send custom metrics via this route.

Therefore to scrape custom metrics I used the Prometheus Python client https://github.com/prometheus/client_python with its own http server start_http_server(8000). I can see these metrics if I port forward to the spark-driver pod on port 8000.

However, I can not find a way to get these custom metrics into Prometheus.

I have tried a PodMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: pod-monitor-myapp-spark-executor
  namespace: stream-myapp
  labels:
    spark-role: driver
    release: prometheus
    serviceMonitorSelector: prometheus
spec:
  selector:
    matchLabels:
       spark-role: driver
  podMetricsEndpoints:
    - port: 8000

my spark operator has the standard Prometheus config set(relevant sections below):

......
  volumes:
  - name: metrics-configmap
    configMap:
      name: spark-prometheus-metrics-properties-myapp
  monitoring:
    exposeDriverMetrics: true
    exposeExecutorMetrics: true
    metricsPropertiesFile: "/etc/metrics/conf/prometheus.metrics.properties/prometheus.metrics.properties"
    prometheus:
      jmxExporterJar: "/mnt/spark/jars-dir/jmx_prometheus_javaagent-0.11.0.jar"
      port: 8080

......
  sparkConf:

    "spark.ui.prometheus.enabled": "true"
    "spark.executor.processTreeMetrics.enabled": "true"

    "spark.metrics.conf": "/etc/metrics/conf/prometheus.metrics.properties/prometheus.metrics.properties"
    "spark.kubernetes.driver.annotation.prometheus.io/scrape": "true"
    "spark.kubernetes.driver.annotation.prometheus.io/path": "/metrics/executors/prometheus/"
    "spark.kubernetes.driver.annotation.prometheus.io/port": "4040"
    "spark.sql.streaming.metricsEnabled": "true"
    "spark.metrics.appStatusSource.enabled": "true"

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-prometheus-metrics-properties-myapp
  namespace: stream-myapp
data:
  prometheus.metrics.properties: |
    *.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
    *.sink.prometheusServlet.path=/metrics/prometheus
    master.sink.prometheusServlet.path=/metrics/master/prometheus
    applications.sink.prometheusServlet.path=/metrics/applications/prometheus
---

I am not sure where to trouble shoot now. Do I need to open port 8000 on the driver pod. I could not find a way to do that on the sparkoperator. What am I missing? Thanks