How to set Preferred Service Name to Spark driver on Kubernetes

Question

I want to setup Ingres to access the Spark Web UI which deployed on Kubernetes cluster.

I created Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: some-test-job-v1
spec:
  ingressClassName: nginx-private
  rules:
    - host: some-test-job-v1.spark.example.com
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service: 
              name: some-test-job-v1-ui-svc
              port:
                number: 4040
  tls:
    - hosts:
        - some-test-job-v1.spark.example.com
      secretName: some-test-job-v1-spark-example-com

The problem is Spark generates service name as spark-[randomServiceId]-driver-svc, so we cant configure Ingress with specified constant service name, like 'some-test-job-v1-ui-svc'. There is a described possibility to set Preferred Service Name, but what property should I pass? Tried spark.kubernetes.resourceNamePrefix: "test", but it still creating with serviceRandomId.

Answer 1

Unfortunately you can't seem to set that property, see related code here and here.

However since Spark 3.2 you can implement custom feature step classes that can execute code after the Driver service is created so you could implement code that would save the service name to a config map for example. Later you could use the value from the config map to initialize your Ingress object. See spark.kubernetes.driver.pod.featureSteps property in https://spark.apache.org/docs/latest/running-on-kubernetes.html.

Note that this is just an idea, I haven't tried it myself.