I have a dag where I am runnnig diff sql queries based on the countries. I want to make an airflow dag where out of 4 queries my 1st query is diff but the other queries are pretty much same.
what I mean by first query diff is that it has diff columns but then the other process does the same job of update, insert etc.
I am little confused how can i do that.
My Process:
- Shortcircuit operator check if there is new data in the table or not.
- If no new data based on timestamp which is stored in s3 directory skip the rest of the dag 3.If new data then based on the country run the dag, I am running my Shortcircuit Operator in for loop for countries countries = ["INDIA","ZAMBIA","USA"]. All of these have diff query but then in second step they all do the same process of update and delete
I used the branch operator but couldn't quite get that correctly.
I want to acheieve something like below image or I may be wrong but suggest
UPDATE: I was able to solve this!