I run Airflow on Windows 11 (Docker Desktop with wsl2 backend) from docker-compose, using LocalExecutor. After all containers are spined up, no DAGs shown on UI, the dags table in database is empty also. Docker-compose from this repo: https://github.com/ClearPeaks/airflow-local-enviroment (change into my company image, which works fine on my Macbook)
I'm sure that:
- DAGs .py files mounted into dags_folder mentioned in airflow.cfg, this dags folder is not empty
- I provide wsl & Docker enough resources using .wslconfig (memory=12GB, processors=8, swap=4GB)
In the scheduler logs: this ERROR occur over and over
DagFileProcessorManager (PID=160) last sent a heartbeat 60.08 seconds ago! Restarting it
In the logs\dag_processor_manager\dag_processor_manager.log
[2024-03-27T04:23:58.122+0000] {manager.py:451} INFO - Exiting gracefully upon receiving signal 15 [2024-03-27T04:23:58.362+0000] {manager.py:469} INFO - Processing files using up to 2 processes at a time [2024-03-27T04:23:58.364+0000] {manager.py:470} INFO - Process each file at most once every 100 seconds [2024-03-27T04:23:58.365+0000] {manager.py:471} INFO - Checking for new files in /opt/airflow/dags every 300 seconds [2024-03-27T04:23:58.365+0000] {manager.py:738} INFO - Searching for files in /opt/airflow/dags [2024-03-27T04:24:06.807+0000] {manager.py:451} INFO - Exiting gracefully upon receiving signal 15
I've found that if Dag file checked, it's will be something like this:
I've try:
- reset factory, modify Airflow scheduler heartbeat configs, follow methods in these links: https://github.com/apache/airflow/discussions/19270 Airflow Sechulder error ERROR - DagFileProcessorManager (PID=1234) last sent a heartbeat 50.72 seconds ago! Restarting it
- reinstall Airflow and wsl
- run Airflow standalone, not using the docker-compose above
- I weird that when I try mount DAGs into an empty folder instead, and add new DAGs to this one-by-one, it works, then will be paused at sometime later.
Nothing works. What's wrong and how can I fix/investigate this? Thank you!!