Issue while running Monthly Cron Expression with Airflow

Question

i need some help understanding the behaviour with monthly Cron expression [43 10 3,8,12 */3 *] with start_date as datetime(year=2019, month=11, day=18, hour=1, minute=30, second=0 , tzinfo=pendulum.timezone("UTC")) and end_date as None . This has backfill set as true .

Current Date is: 2020-10-19

As per my understanding it should not have triggered last two runs 10-03 and 10-08 . Can someone please help me understand this behavior? Also if it is triggering run for execution_date of 10-03 and 10-08 then why not for 10-12?

Answer 1

Could you elaborate on "it should not have trifggered the last two runs"?

The cron expression 43 10 3,8,12 */3 * matches:

“At 10:43 on day-of-month 3, 8, and 12 in every 3rd month.”

A good tool to validate cron expression is crontab.guru.

The execution date 10-12 hasn't triggered yet, because of how Airflow handles execution_date - see airflow scheduler:

The scheduler won’t trigger your tasks until the period it covers has ended e.g., A job with schedule_interval set as @daily runs after the day has ended. This technique makes sure that whatever data is required for that period is fully available before the dag is executed. In the UI, it appears as if Airflow is running your tasks a day late

Let’s Repeat That, the scheduler runs your job one schedule_interval AFTER the start date, at the END of the period.

This means the run with execution date 2020-10-12 10:43:00 will be triggered just shortly before 2021-01-03 10:43:00.