We are trying to find out why our Airflow Scheduler is making so many system calls to the DAG files?
We have Airflow v2.5.3 deployed in AWS with the Scheduler and Webserver on the same EC2 instance and multiple Workers on their own EC2 instances. They all share an EFS provided NFS mount providing a shared configuration setup (i.e. they all use the same airflow.cfg, DAGs Directory, and Python Environment).
We noticed our EFS charges increasing (due to NFS bandwidth usage). We used strace to determine that it was the Airflow Scheduler constantly polling the DAG files. When I say constantly, I mean each DAG is being polled via newfstatat twice every second. With 63 DAGs that is 126 calls per second (for no apparent reason).
In our airflow.cfg file we have the following two settings:
# min_file_process_interval set for every 2 minutes
min_file_process_interval = 120
# dag_dir_list_interval set for every 5 minutes
dag_dir_list_interval = 300
We could see in the strace logs precisely when the scheduler was initiating both of the above tasks as there were multiple child processes spawned to handle the DAG file parsing in response to min_file_process_interval and appropriate lines showing openat calls in sync with the dag_dir_list_interval timing. This is all well and good and was to be expected. However, in between these expected events are the constant newfstatat calls to each DAG file.
The strace command we ran was the following:
sudo strace -f -t -e trace=file -p <PID of Airflow Scheduler - DagFileProcessorManager> -Y -yy -T -o strace.log
A sampling of the strace.log output for a roughly one second period (showing the constant system calls to newfstatat) is as follows:
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/DAG-Number-1.py", {st_mode=S_IFREG|0664, st_size=8640, ...}, 0) = 0 <0.000054>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU6-Zabbix-Data-To-Snowflake.py", {st_mode=S_IFREG|0664, st_size=8740, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/BAC_invoicing.py", {st_mode=S_IFREG|0664, st_size=17227, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-VPI-Create-Tickets.py", {st_mode=S_IFREG|0664, st_size=6576, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-VPI-Alert-Manager.py", {st_mode=S_IFREG|0664, st_size=2338, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-Remove-User.py", {st_mode=S_IFREG|0664, st_size=14773, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Teams-Migration-Data-Validation.py", {st_mode=S_IFREG|0664, st_size=8339, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-AURA-Teams-Phone-Migration.py", {st_mode=S_IFREG|0664, st_size=4989, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-SBC-DialPlan.py", {st_mode=S_IFREG|0644, st_size=3471, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-One-X-Software.py", {st_mode=S_IFREG|0664, st_size=5219, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-EC500-Equinox.py", {st_mode=S_IFREG|0664, st_size=6839, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Teams-Migration.py", {st_mode=S_IFREG|0664, st_size=14682, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-CLI-Ctrl-Test.py", {st_mode=S_IFREG|0664, st_size=10907, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-Desktop-Phone.py", {st_mode=S_IFREG|0664, st_size=20267, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Avaya.py", {st_mode=S_IFREG|0664, st_size=9181, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-VMWare.py", {st_mode=S_IFREG|0664, st_size=8904, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Cert.py", {st_mode=S_IFREG|0664, st_size=8865, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-HP.py", {st_mode=S_IFREG|0664, st_size=8825, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Suse.py", {st_mode=S_IFREG|0664, st_size=8863, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Microsoft-NVD.py", {st_mode=S_IFREG|0644, st_size=11018, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Mitel.py", {st_mode=S_IFREG|0664, st_size=8886, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Microsoft-Feed.py", {st_mode=S_IFREG|0664, st_size=8743, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-RSS-Bund-WID.py", {st_mode=S_IFREG|0664, st_size=8706, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Cisco.py", {st_mode=S_IFREG|0664, st_size=8755, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Dell.py", {st_mode=S_IFREG|0664, st_size=8865, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-Vendor_Release_Detail-Workflow.py", {st_mode=S_IFREG|0664, st_size=12133, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-PRTG-Watchdog_Metric_Collector.py", {st_mode=S_IFREG|0664, st_size=4309, ...}, 0) = 0 <0.000049>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-OT-Ticket-Manager-Workflow.py", {st_mode=S_IFREG|0664, st_size=16262, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-PRTG-Watchdog.py", {st_mode=S_IFREG|0664, st_size=4443, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-OT-NPS-Workflow.py", {st_mode=S_IFREG|0664, st_size=22145, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-PRTG-Workflow.py", {st_mode=S_IFREG|0664, st_size=13019, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Create-Extension.py", {st_mode=S_IFREG|0664, st_size=13128, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Name-Change.py", {st_mode=S_IFREG|0664, st_size=6140, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Check-credentials.py", {st_mode=S_IFREG|0664, st_size=10817, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Delete-Extension.py", {st_mode=S_IFREG|0664, st_size=7825, ...}, 0) = 0 <0.000046>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Update_Overlord_Keys.py", {st_mode=S_IFREG|0664, st_size=2379, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Monitor.py", {st_mode=S_IFREG|0664, st_size=3383, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Update_Overlord_Capabilities.py", {st_mode=S_IFREG|0664, st_size=2643, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overseer-Overlord_Persistence.py", {st_mode=S_IFREG|0664, st_size=2808, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Teams-Test.py", {st_mode=S_IFREG|0664, st_size=3765, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overseer-Collection_Parser.py", {st_mode=S_IFREG|0664, st_size=2015, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-BitBucket.py", {st_mode=S_IFREG|0664, st_size=2920, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-gto_backup.py", {st_mode=S_IFREG|0664, st_size=9056, ...}, 0) = 0 <0.000063>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_backup_configs.py", {st_mode=S_IFREG|0664, st_size=9182, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_clear_missing_dags.py", {st_mode=S_IFREG|0664, st_size=4618, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_kill_halted_tasks.py", {st_mode=S_IFREG|0664, st_size=20250, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-find_long_running_tasks.py", {st_mode=S_IFREG|0664, st_size=6182, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_log_cleanup_PRTG.py", {st_mode=S_IFREG|0664, st_size=11042, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_log_cleanup.py", {st_mode=S_IFREG|0664, st_size=9093, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_delete_broken_dags.py", {st_mode=S_IFREG|0664, st_size=3216, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Name-Change.py", {st_mode=S_IFREG|0664, st_size=9084, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-CXE-Deactivate-Voicemail.py", {st_mode=S_IFREG|0664, st_size=4490, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Delete-Onephone.py", {st_mode=S_IFREG|0664, st_size=4362, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Create-Extension.py", {st_mode=S_IFREG|0664, st_size=8759, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Delete-Extension.py", {st_mode=S_IFREG|0664, st_size=8574, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Create-Onephone.py", {st_mode=S_IFREG|0664, st_size=6709, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Check-credentials.py", {st_mode=S_IFREG|0664, st_size=10785, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-AURA-Name-Change.py", {st_mode=S_IFREG|0664, st_size=10718, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Error-271.py", {st_mode=S_IFREG|0644, st_size=9422, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-VPN-Connect-Atmos-Agent.py", {st_mode=S_IFREG|0664, st_size=10676, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Error-349.py", {st_mode=S_IFREG|0664, st_size=9295, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Error-268.py", {st_mode=S_IFREG|0664, st_size=9422, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-AURA-Create-Extension.py", {st_mode=S_IFREG|0664, st_size=6055, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/DAG-Number-1.py", {st_mode=S_IFREG|0664, st_size=8640, ...}, 0) = 0 <0.000048>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU6-Zabbix-Data-To-Snowflake.py", {st_mode=S_IFREG|0664, st_size=8740, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/BAC_invoicing.py", {st_mode=S_IFREG|0664, st_size=17227, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-VPI-Create-Tickets.py", {st_mode=S_IFREG|0664, st_size=6576, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-VPI-Alert-Manager.py", {st_mode=S_IFREG|0664, st_size=2338, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-Remove-User.py", {st_mode=S_IFREG|0664, st_size=14773, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Teams-Migration-Data-Validation.py", {st_mode=S_IFREG|0664, st_size=8339, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-AURA-Teams-Phone-Migration.py", {st_mode=S_IFREG|0664, st_size=4989, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-SBC-DialPlan.py", {st_mode=S_IFREG|0644, st_size=3471, ...}, 0) = 0 <0.000063>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-One-X-Software.py", {st_mode=S_IFREG|0664, st_size=5219, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-EC500-Equinox.py", {st_mode=S_IFREG|0664, st_size=6839, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Teams-Migration.py", {st_mode=S_IFREG|0664, st_size=14682, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-CLI-Ctrl-Test.py", {st_mode=S_IFREG|0664, st_size=10907, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU1/CU1-Aura-Desktop-Phone.py", {st_mode=S_IFREG|0664, st_size=20267, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Avaya.py", {st_mode=S_IFREG|0664, st_size=9181, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-VMWare.py", {st_mode=S_IFREG|0664, st_size=8904, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Cert.py", {st_mode=S_IFREG|0664, st_size=8865, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-HP.py", {st_mode=S_IFREG|0664, st_size=8825, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Suse.py", {st_mode=S_IFREG|0664, st_size=8863, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Microsoft-NVD.py", {st_mode=S_IFREG|0644, st_size=11018, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Mitel.py", {st_mode=S_IFREG|0664, st_size=8886, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Microsoft-Feed.py", {st_mode=S_IFREG|0664, st_size=8743, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-RSS-Bund-WID.py", {st_mode=S_IFREG|0664, st_size=8706, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Cisco.py", {st_mode=S_IFREG|0664, st_size=8755, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Vulnerability/Vulnerability-Dell.py", {st_mode=S_IFREG|0664, st_size=8865, ...}, 0) = 0 <0.000046>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-Vendor_Release_Detail-Workflow.py", {st_mode=S_IFREG|0664, st_size=12133, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-PRTG-Watchdog_Metric_Collector.py", {st_mode=S_IFREG|0664, st_size=4309, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-OT-Ticket-Manager-Workflow.py", {st_mode=S_IFREG|0664, st_size=16262, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-PRTG-Watchdog.py", {st_mode=S_IFREG|0664, st_size=4443, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-OT-NPS-Workflow.py", {st_mode=S_IFREG|0664, st_size=22145, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU2/CU2-PRTG-Workflow.py", {st_mode=S_IFREG|0664, st_size=13019, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Create-Extension.py", {st_mode=S_IFREG|0664, st_size=13128, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Name-Change.py", {st_mode=S_IFREG|0664, st_size=6140, ...}, 0) = 0 <0.000051>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Check-credentials.py", {st_mode=S_IFREG|0664, st_size=10817, ...}, 0) = 0 <0.000049>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU3/CU3-MX-ONE-Delete-Extension.py", {st_mode=S_IFREG|0664, st_size=7825, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Update_Overlord_Keys.py", {st_mode=S_IFREG|0664, st_size=2379, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Monitor.py", {st_mode=S_IFREG|0664, st_size=3383, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Update_Overlord_Capabilities.py", {st_mode=S_IFREG|0664, st_size=2643, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overseer-Overlord_Persistence.py", {st_mode=S_IFREG|0664, st_size=2808, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-Teams-Test.py", {st_mode=S_IFREG|0664, st_size=3765, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overseer-Collection_Parser.py", {st_mode=S_IFREG|0664, st_size=2015, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU4/CU4-Overmind-BitBucket.py", {st_mode=S_IFREG|0664, st_size=2920, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-gto_backup.py", {st_mode=S_IFREG|0664, st_size=9056, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_backup_configs.py", {st_mode=S_IFREG|0664, st_size=9182, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_clear_missing_dags.py", {st_mode=S_IFREG|0664, st_size=4618, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_kill_halted_tasks.py", {st_mode=S_IFREG|0664, st_size=20250, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-find_long_running_tasks.py", {st_mode=S_IFREG|0664, st_size=6182, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_log_cleanup_PRTG.py", {st_mode=S_IFREG|0664, st_size=11042, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_log_cleanup.py", {st_mode=S_IFREG|0664, st_size=9093, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/Maintenance/CU4-Maintenance-airflow_delete_broken_dags.py", {st_mode=S_IFREG|0664, st_size=3216, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Name-Change.py", {st_mode=S_IFREG|0664, st_size=9084, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-CXE-Deactivate-Voicemail.py", {st_mode=S_IFREG|0664, st_size=4490, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Delete-Onephone.py", {st_mode=S_IFREG|0664, st_size=4362, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Create-Extension.py", {st_mode=S_IFREG|0664, st_size=8759, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Delete-Extension.py", {st_mode=S_IFREG|0664, st_size=8574, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Create-Onephone.py", {st_mode=S_IFREG|0664, st_size=6709, ...}, 0) = 0 <0.000045>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Check-credentials.py", {st_mode=S_IFREG|0664, st_size=10785, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-AURA-Name-Change.py", {st_mode=S_IFREG|0664, st_size=10718, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Error-271.py", {st_mode=S_IFREG|0644, st_size=9422, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-VPN-Connect-Atmos-Agent.py", {st_mode=S_IFREG|0664, st_size=10676, ...}, 0) = 0 <0.000043>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Error-349.py", {st_mode=S_IFREG|0664, st_size=9295, ...}, 0) = 0 <0.000044>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-MX-ONE-Error-268.py", {st_mode=S_IFREG|0664, st_size=9422, ...}, 0) = 0 <0.000046>
52603<airflow schedul> 19:39:18 newfstatat(AT_FDCWD</air1/airflow>, "/air1/airflow/dags/CU5/CU5-AURA-Create-Extension.py", {st_mode=S_IFREG|0664, st_size=6055, ...}, 0) = 0 <0.000044>
As a result of these constant calls (which I assume are probably just looking at the mtime on each file) our EFS/NFS Bandwidth usage is extremely high. Looking in the AWS Console at the EFS Monitoring for this particular share we can see Metadata outweighing both Reads and Writes by a disproportionate amount (i.e. METADATA taking 98% of the bandwidth, READS taking 1.95%, and WRITES taking 0.05 ).
For testing and to relive the bandwidth usage we copied the DAGs locally so the scheduler could query locally rather than across the NFS mount. As soon as we did this the NFS usage dropped and so did the excessive bandwidth charges. But still, using strace we can see the continuous calls occurring (just locally now).
So does anyone have any idea what is going on here? I checked the Airflow Release Notes to see if there was any bug/fix that may address this issue (up to v2.7.3) but saw nothing. I really see no reason why the scheduler needs to be constantly hitting the DAG files (outside of the two configuration options presented above).
For now I am going to look into using rsync to keep the local DAG files in sync with the remote DAG files for now. But hoping there may be something (maybe a configuration setting) that I can tweak to remedy this situation.
Any help would be appreciated!
Russell