Apache spark with on prem s3 integration

Question

I am using spark with on prem s3 (Minio) and spark operator. So when we are using spark with s3 (without enabling ssl ). It is working fine. We were able to get data, write data and can run our main application jar present in s3 also. However, when enabling ssl and providing self trust certificate(for now). We are start facing some issue.

1. When we trying to test spark connectivity with s3. We ran local job (master= local[]) and tried to provide our ssl by adding it to the jks truststore. which resolve our issue.

here is command

./spark-submit \
--master local[*] \
--name ml-pipeline-16 \
--conf spark.ssl.server.keystore.type=jks \
--conf spark.ssl.server.keystore.password=changeit \
--conf spark.ssl.server.keystore.location=/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/security/cacerts \
--conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.keyStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_281.jdk/Contents/Home/jre/lib/security/cacerts -Djavax.net.ssl.keyStorePassword=changeit" \
--class com.abc.dp.efg.DataSetGenerator \
/Users/ayush.goyal/IdeaProjects/test/target/SparkS3-1.0-SNAPSHOT-jar-with-dependencies.jar 

2. When we tried to run it by providing application jar in s3 itself it is not able to connect to s3 to start the job because we are providing certificate in sparkconf parameter and getting below error.

Exception in thread "main" org.apache.hadoop.fs.s3a.AWSBadRequestException: doesBucketExist on minio21: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: null; S3 Extended Request ID: null; Proxy: null)

here is our spark operator yaml

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: minio-stg-test-4
spec:
  type: Java
  mode: cluster
  image: "stg-test/spark/spark-py:v3.1.1-h3-2"
  imagePullPolicy: IfNotPresent
  mainClass: com.abc.dp.efg.DataSetGenerator
  mainApplicationFile: "s3a://minio21/jars/SparkS3-1.0-SNAPSHOT-jar-with-dependencies.jar"
  sparkVersion: "3.1.1"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/tmp"
        type: Directory
  driver:
    configMaps:
    - name: minio-certificate
      path: /mnt/config-maps
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.1.1
    serviceAccount: spark-user
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  executor:
    configMaps:
    - name: minio-certificate
      path: /mnt/config-maps
    cores: 1
    instances: 2
    memory: "512m"
    labels:
      version: 3.1.1
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  sparkConf:
    "spark.kubernetes.file.upload.path": "s3a://minio21/tmp"
    "spark.hadoop.fs.s3a.access.key": "fdgvbsgt"
    "spark.hadoop.fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem"
    "spark.hadoop.fs.s3a.fast.upload": "true"
    "spark.hadoop.fs.s3a.secret.key": "sfbdfbbsdrbh44q3#$"
    "spark.hadoop.fs.s3a.endpoint": "http://[278b:c1r0:0012:5ed3:b112:2::]:30000"
    "spark.hadoop.fs.s3a.path.style.access": "true"
    "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName": "OnDemand"
    "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.storageClass": "robin"
    "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.sizeLimit": "200Gi"
    "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path": "/tmp/spark-local-dir"
    "spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly": "false"
    "spark.executor.extraJavaOptions": "-Djavax.net.ssl.keyStore=/mnt/config-maps/cacerts -Djavax.net.ssl.keyStorePassword=changeit"
    "spark.ui.port": "4041"
    "spark.ssl.server.keystore.type": "jks"
    "spark.ssl.server.keystore.password": "changeit"
    "spark.ssl.server.keystore.location": "/mnt/config-maps/cacerts"

Note : Earlier we were using s3 by disabling ssl and we were able to run our job by providing application jar in s3 like we did in above yaml.

How can we run our job like trying to do? is it possible?