Context:
- Using Google Cloud Composer to schedule and orchestrate some tasks.
- There are around 20 active DAGs currently. It will grow over time (around 40-50).
- Each DAG has 4 tasks. They are scheduled to run every hour.
- Each task is a dbt run command. Every task selects specific set of models.
- Each task typically take around 1-5 mins depending on the selected models. They are not very resource intensive. Total DAG run shouldnt take more than 10-15 mins.
My problem is that the workers not scaling even though there are 100s of tasks queued in celery.
As you can see below, tasks keep getting queued but the worker is not increasing.
This is my resource configuration:
And this is my only Airflow configuration that I have overriden:
What am I doing wrong here? Why is the active worker 0 even though minimum is 1? I know I dont need more resource per worker to execute the task because the tasks used to get executed before.
I have seen this answer already. It suggested to reduce the worker_concurrency (which is already small in my case) and increase the max_active_tasks_per_dag (this setting is useless because tasks need to run sequentially in my DAGs)
According to the documentation, it uses Scale Factor Target metric to scale the workers. I've checked the metric, and it keeps on increasing but the workers dont scale. It stays at 0 for some reason!
