ADF single event trigger for multiple files

Question

I have configured event trigger for blob storage folder like container/daily_files . Let us say file1.txt and file2.txt will land simultaneously so it will trigger two instances of the same pipeline. I want to control this and execute with only single instance of ADF pipeline. Is there any way we can achieve this ?

Answer 1

Unfortunately there is no direct way. You would have to either :

1. Make the parallel execution of the pipeline 1, so that even if the pipeline is triggered twice, the execution would be sequential. And then process all files in initial execution, archive them and in 2nd execution skip the execution. somewhat similar in my blog: https://datasharkx.wordpress.com/2022/09/24/event-trigger-of-synapse-azure-data-factory-pipeline-on-arrival-of-nth-file-in-azure-blob-storage/

2. The blob trigger can be on a pipeline which can in turn trigger your main pipeline based on validating the file count

Answer 2

AFAIK there is no way to control pipeline run once it is got triggered simultaneously.

You can set pipeline Concurrency to 1. It will be act like it would start pipeline run for the first trigger if pipeline got triggered in between the first run it will add second run in queue.

Note - Please be aware that there is a limit on the size of the queue. Using pipeline concurrency of 1, for example, might result in one pipeline run being in progress and 100 queue entries; if this happens and the pipeline continues to trigger, error 429 will occur.