Kubernetes pod operator volume mount is failing, resulting in pods not being spawned

Question

I am trying to convert a spark-submit job which was being configured through yaml file to a airflow dag and run it through airflow. The job is failing when I try to mount a volume and load the config-map. The volumne mount block of code is something like this in the yaml

  volumeMounts:
    - mountPath: /etc/jtp
      name: config-vol

And the config-map which is to be loaded is something like this

  volumes:
  - configMap:
      name: qa-jtp-llt
    name: config-vol

Now when I try to translate the same thing to my dag it transforms to something like this

from kubernetes.client import models as k8s
from airflow import DAG
from datetime import timedelta
from datetime import datetime
from airflow.utils.dates import days_ago
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.utils.trigger_rule import TriggerRule


volume_mount = k8s.V1VolumeMount(name='config-vol', mount_path='/etc/jtp', sub_path=None, read_only=True)

configmaps = [k8s.V1EnvFromSource(config_map_ref=k8s.V1ConfigMapEnvSource(name='config_name'))]

volume = k8s.V1Volume(
    name='config-vol',
    persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(claim_name='config-vol'),
)

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'wait_for_downstream': False,
    'start_date': datetime(2020, 10, 8),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'max_active_runs': 1,
    'catchup': True
}   a


dag = DAG(
    'sample_job',
    default_args=default_args,
    description='sample job to run on kubernetes-pod-operator',
    schedule_interval=None,
)

sample_job = KubernetesPodOperator(
        task_id='task-1',
        name='sample-job',
        namespace='namespace1',
        image='image_name',
        image_pull_policy='Always',
        volumes=[volume],
        volume_mounts=[volume_mount],
        configmaps=configmaps,
        cmds=["/opt/entrypoint.sh", "driver"],
        arguments=[
            '--class', 'something.thisthing.mainclass',
            'myjar_file_name.jar',
            'property_file_1.properties',
            'property_file_2.properties',
        ],
        dag=dag,
    )

I am trying this out on airflow 2.0.0 version and using kubernetes pod operator. The problem I am facing is when I run this job on Airflow this in turn starts a local pod for the job which should in turn start another pod in the namespace running the task. But the second pod which should run the task doesn't start and there is no error message. When I remove all the components of the volume specification such as volume, volume_mount, config_map then the second pod in the namespace spawns and gives the error message of not able to find the property file. What am I missing ?? What am I doing wrong here ?? Tried various ways of starting it failed so turning towards this community for help.

Thanks

Answer 1

Most likely you don't have pvc with name "config-vol" in "namespace1" namespace.

You need to provide the correct PVC name for V1VolumeMount(name='existing-pvc-name',...)

You should be able to list available pvc's in that namespace with:

kubectl -n namespace1 get pvc