Autoscaling Google Cloud Composer

Question

I have read this Medium article which is one of the top hits when searching for autoscaling and Cloud Composer. It shows some 'hacks' that you can use to autoscale Composer while it remains configured to use the CeleryExecutor.

I have also read the GCP docs on using KubernetesPodOperator (KPO) with Cloud Composer, and have implemented that before.

However, using KPO means that you don't get to utilize all the other Airflow Operators - you have to write your own container and code every time.

KubernetesExecutor seems to be the best way forward - you get to use the Airflow Operators, and autoscaling can be enabled since it will create a new Kubernetes Pod for every task instance.

Google Cloud Composer currently runs on CeleryExecutor, in Blocked Airflow Configurations it currently states

Cloud Composer configures Airflow to use the Celery executor for the core-executor setting.

Will KubernetesExecutor ever be an option for Composer?

Answer 1

I would recommend this airflow-executors-explained overview for features comparison of CeleryExecutor and KubernetesExecutor. As you have already researched in the links you have provided, the CeleryExecutor does provide the scalability for the Composer environment. Having KubernetesExecutor as an option would be nice, but is not necessary as the additional benefits don't outweigh the downsides.

Answer 2

Unfortunately, your question can't be answered yet as there is no official plans of doing so. That said, I would be surprised if this wasn't at least under consideration by the Cloud Composer product team.
But as soon as there is news about it, it should be published in this Feature Request.