DBT (Data Build Tools) - Create CI/CD pipeline in Azure DevOps

Question

I am working on a project that uses DBT by Fishtown Analytics for ELT processing. I am trying to create a CI/CD pipeline in Azure DevOps to automate the build release process, but I am unable to find a suitable documentation around it. The code has been integrated in DevOps Repos, now I need a reference to start with building the CI/CD pipelines.

Answer 1

Our team uses ADO Pipelines, here’s our doc on how we do it!

To make this example work, you will need 3+ files:

1. profiles.yml.

If you look at this file, you'll see usage of dbt-jinja's env_var macro. This lets you make ADO Pipeline secret variables for your database credentials, and make them available to dbt.

2. step_template.yml

This file is a recipe of deployment steps that are used when deploying to different environments. The beauty of the dbt CLI is that the same steps are used when:

• doing our integration tests (gatekeeper), and
• deploying to our production and staging/UAT environments. So this step_template.yml lets each pipeline re-use the same steps. It's worth noting here that because we are using the AzureCLI@2 task because it will auto-authenticate to Azure for us. If you're not using Azure, you'll need to:
• add Pipeline secrets for username and password to the db, and
• use the Bash@3 task

3. gatekeeper.yml and prod.yml

These are the actual pipelines. If you look, they are identical except for:

• the variables they deploy to different databases), and
• the trigger changes to which branch will trigger this pipeline?
• the schedule should this run even if there are no new commits to the branch? Yes, if production because we want the data to stay up-to-date for our users.