How can I pass parameters to databricks.yml in Databricks Asset Bundles?

Question

Background: I have a separate Databricks Workspace for each environment, and I am buidling an Azure DevOps pipeline to deploy a Databricks Asset Bundles to these environments.

Question The asset bundle is configured in a databricks.yml file. How do I pass parameters to this file so I can change variables depending on the environment?

For example, I want to be able to vary my scheduling such that it runs more often in Prod than in the lower level environments. In the code below I want to pass {cronExpression} to the file from the pipeline.

Example from databricks.yml:

bundle:
  name: bundle_name

resources:
  jobs:
    job_name:
      name: job_name
      schedule:
        quartz_cron_expression: {inputParameter}
      tasks:
        ...

I have looked at Custom Variables, but it seems they need to be hard-coded in databricks.yml. https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/settings#--custom-variables

Answer 1

In the code below I want to pass {cronExpression} to the file from the pipeline.

In DevOps pipeline, you can use replacetokens@5 to dynamically set the value in file.

For example, have the databricks.yml below, notice the #{inputParameter}# format as it will be evaluated in task.

In DevOps, install extension Replace Tokens, add task as below：

I add the variable inputParameter i want to use.

I output the yml file for confirmation:

Answer 2

There are a few methods of passing a value of a variable - it's all described in the documentation:

• specify it at the top-level of the databricks.yaml - then it will be the same for all environments (if they don't override it).
• you can override them for each environment/target separately
• you can pass them via environment variables with names like BUNDLE_VAR_<variable_name>
• pass them via command-line: --var="<var-name>=<var-value>"

But anyway you need to declare that variable before use - otherwise, the bundle validate command won't be able to resolve it & perform validation.

If you want to do that from the Azure DevOps, then the last method could be the best, or maybe environment variables.