Schedule a DAG at different time by using Airflow REST API with conf variables

Question

I want to scrape a website daily by using AirFlow and Selenium Grid server. I have created a single DAG for this.

The website provide an endpoint (e.g. website.com/python, website.com/java, website.com/ruby) for each data in a same web user interface.

I've succesfully triggered The DAG by using AirFlow REST API with a conf variable. Now I want to schedule every run, which I have triggered, daily with the coressponding conf variable.

Is it possible to achieve this (maybe by creating a new DAG or dynamically set a cronjob) ?

Answer 1

You can schedule an Airflow DAG by setting schedule on the DAG, for example:

with DAG(dag_id="hello_world", start_date=datetime(2023, 1, 1), schedule="@daily"):
    ...

From your question, it's unclear to me if your conf changes per DAG run. If not, you could hardcode all the settings, or set them through params on the DAG, see example here.