Is there an api to send notifications based on job outputs?

Question

I know there are api to configure the notification when a job is failed or finished.

But what if, say, I run a hive query that count the number of rows in a table. If the returned result is zero I want to send out emails to the concerned parties. How can I do that?

Thanks.

Answer 1

There is no direct api for triggering notification based on results of a query. However there is a way to do this using Qubole: -Create a work flow in qubole with following steps: 1. Your query (any query) that writes output to a particular location on s3. 2. A shell script - This script reads result from your s3 and fails the job based on any criteria. For instance in your case, fail the job if result returns 0 rows. -Schedule this work flow using "Scheduler" API to notify on failure.

You can also use "Sendmail" shell command to send mail based on results in step 2 above.

Answer 2

You may want to look at Airflow and Qubole's operator for airflow. We use airflow to orchestrate all jobs being run using Qubole and in some cases non Qubole environments. We DataDog API to report success / failures of each task (Qubole / Non Qubole). DataDog in this case can be replaced by Airflow's email operator. Airflow also has some chat operator (like Slack)