I am using Airflow2.0 with KubernetesPodOperator
want to run a command that use as a parameter a file from inside the image ran by the Operator. This is what I used:
KubernetesPodOperator(
namespace=commons.kubernetes_namespace,
labels=commons.labels,
image=f"myregistry.io/myimage:{config['IMAGE_TAG']}",
arguments=[
"python",
"run_module.py ",
"-i",
f'args/{config["INPUT_DIR"]}/{task_id}.json'
],
name=dag_name + task_id,
task_id=task_id,
secrets=[secret_volume]
)
But this gives me the error:
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: args/airflow2test/processing-pipeline.json
The image does not use any macros.
Anyone has any clue? What do I do wrong?
This was a bug that started with PR released in version
2.0.0
ofapache-airflow-providers-cncf-kubernetes
. The goal of the change was to allow templating of.json
files. There was a GitHub issue about the problems it created. The bug was eventually resolved by PR which was released in version 2.0.2 of the provider.Solution:
apache-airflow-providers-cncf-kubernetes
(currently 2.0.2)KubernetesPodOperator
There are two ways to workaround that problem one is to change
template_fields
the other is to changetemplate_ext
:1st option: As posted on issue by raphaelauv is not to allow rendering of
arguments
field:2st option: if you prefer not to render
.json
files: