Azure Databricks - Long running processes

Question

All,

I'm in the process of learning Databricks. If there is a runaway process that has been running a Job Cluster, I would like to get notified/alerted and if possible event terminated. In SQLServer, this is normally done with the help of Query Governor.

IS there something comparable in Azure Databricks that I put to use?

Thanks, grajee

Answer 1

Databricks jobs (or new name - workflows) support sending the email alerts on jobs start and/or termination (successful or errors only) - just use UI or API to configure these notifications. Also you can configure such notifications on the level of individual tasks inside the workflow/job

Answer 2

You can add one or more email addresses to notify when runs of this job begin, complete, or fail:

1. Click Edit alerts.
2. Click Add.
3. Enter the email address to notify and click the check box for each alert type to send to that address.
4. To enter another email address for notification, click Add.
5. If you do not want to receive alerts for skipped job runs, click the check box.
6. Click Confirm.