Multiple activities inside dynamic content in ADF

Question

I have the following pipeline in ADF:

(https://stackoverflow.com/image.jpg)

I have two activities pointing to a Failure SP which has the following parameters:

It's currently set to print out an Error message if the Databricks Notebook fails, but I would like for it to also print an Error message if the Synapse_SP activity fails after the Notebook activity succeeds. Is it possible to add dynamic content to choose the error message from the activity that fails?

The current dynamic content is @activity('Notebook').Error.message

Answer 1

In ADF, the activity flow will be like an AND operator. Here, your Failure SP activity will only be executed when both Notebook and Synapse_SP activities fails which won't happen as your flow is a sequential flow.

If your Notebook activity fails, then the flow won't execute the Synapse_SP activity because it is after the success of Notebook activity. That means Failure activity also won't execute as it is an AND(both activity failure). So, your pipeline flow will be stopped at Notebook activity failure itself.

To achieve your requirement, you can follow two approaches.

One is you need to use On Complete and On skipped outcomes of the activities to obtain the OR of the activity outcomes.

Go through this blog by @Nandan Hegde to understand about this approach.

Another approach would be using two pipelines with Execute pipeline activity. One is a master and other is a child.

Build your master pipeline like this.

Give your lookup activity in the parent pipeline- and use Execute pipeline activity to call the child pipeline. Give the activities which you want to check the errors in the child pipeline.

Here, instead of Notebook and SP activities, I have used two set variable activities with errors.

In the Execute pipeline activity, check the Wait on completion checkbox.

Here, we are handling the errors of all activities in the child pipeline by a single Execute pipeline activity in the parent pipeline. If this child pipeline activity had any error, it stops the flow and Execute pipeline activity also gives the same error.

So, get that failed activity name and error message by using the below expressions.

Take a string type set variable activity on the failure of the Execute pipeline and use below expression to get the failed activity name.

@substring(activity('Execute Pipeline1').error.message,  add(indexOf(activity('Execute Pipeline1').error.message,'target'),7),  sub(indexOf(activity('Execute Pipeline1').error.message,'failed'),add(indexOf(activity('Execute Pipeline1').error.message,'target'),8)))

To get the error message use below expression.

@split(activity('Execute Pipeline1').error.message,'failed: ')[1]

These variables will give the outputs like below:

Use your Failed Stored procedure activity after these and pass the above variables as per your requirement.

You can give your Success Stored procedure activity on the success of the Execute pipeline activity and continue your logic from there.