How to write new DAG with DataBricks API

Question

Using the DataBricks API from Python, how can I clone a template DAG with some minor replacements?

For example:

A template DAG contains some fields named like “template” and in the clone I want then to be “stage”. The newly cloned DAG and its tasks would have the same settings including dependencies.

Answer 1

You just need to use Get Single Job API call to obtain a configuration of a given job, manipulate job's configuration is is returned as a settings block to change necessary parameters, and then create a new job with Create Job API using the settings payload.

But instead of using APIs directly it's better to adopt new Databricks SDK for Python, that simplifies working with APIs, providing authentication, etc. There are multiple examples of using SDK in its documentation.