Design advice calling Cloud Build from Airflow to execute a python function that takes args

Question

I’m seeking some design advice and any examples using Cloud Build to execute a python function that takes args.

I have a data pipeline using Airflow on GCP. Some files need cleaning before loading into BigQuery. I have a Python function that does the job. What I would like is to run that function on a Cloud Build image and pass the function args from my pipeline.

So far I have a Cloud Build trigger that installs and runs a simple Python function with no args.

From my research I have found two options, the cloudbuild_v1 library and a Cloud Build Airflow operator.

I would like advice on which option I should use or if other options should be considered and any example to work from.

Thank you.

Answer 1

There is not just one correct or best option of which one of them you should use. It actually would really depend on your application, your preferences and requisites, etc. I would say for you to take a deeper look at both of them and confirm which one fits you better. If I would say one of them, I would say Cloud Build Airflow, as it has more content available for learning and searching.

Regarding to examples, you can check them here for the cloudbuild_v1 library and here or here for Cloud Build Airflow.