How to select first 10 files out of 47 from ADLS using ADF pipeline

Question

I have 47 files in ADLS folder. I need to batch process it. Every iteration should process 10 files. So it should have 5 iteration with 10 files for each iteration and last iteration with 47 files. I need to implement this logic in ADF pipeline

I tried to filter using the filter activity but not able to filter based on the number of files

Answer 1

To files with the batches, you need to use multiple activities or increment variables to get the batch of 10 file names.

Follow below steps to achieve your requirements:

• First take get meta data to get the list of files from the folder:
• Then take Foreach activity and pass the output of get metadata to foreach activity:
• Then under for each activity take append variable to store all file names in one array:
• After this create two variables with name skip and skipinc type integer and default value as 0
• Now take until activity and set the expression as @equals(variables('skipinc'), 50)
• Under until activity take set variable add variable skip which we created earlier and set the value as @variables('skipinc')
• Then take another set variable to create batches of files using expression @take(skip(variables('files'),variables('skip')),add(variables('skip'),10))

Note: After this step you can take execute pipeline and pass ths variable to that pipeline and process that batch of files

• Then take another set variable to increment the skip value with @add(variables('skip'),10 ).

Output:

For every iteration of until loop it will create a batch of 10 files: