My requirements is to schedule the Airflow Dag weekly/monthly. I have gone through the concepts of airflow like, it will execute the dag once the interval is completed, date_interval_end=date_interval_start+ interval.
In my case I am getting files every weekly on monday at 6:00 am CST in S3 Bucket, I want to process them at 6:30am CST.
eg: I am getting files on 2023-11-13, so I have set start_date as 2023-11-6 and cron expression in scheduler as '30 6 * * 1'. date_interval_start = 2023-11-06 06:30:00 date_interval_end = 2023-11-13 06:30:00.
Appreciate you help, thanks
The DAG is getting triggered but its logical_date is set to 2023-11-06 and its processing last week files instead of the 2023-11-13 files. Overall DAG is processing 1 week old files.
In the code I am making use of ds macro to pick the latest files. I can't set ds to date_interval_end macro as by doing this, the DAG can't be executed for custom date incase of delay in the files.