Azure. StorageAccounts, Azure data factory (ADF). Organization of the folders and subfolders

Question

I was wondering if the way me and my group are storing the files is a best practice, if it is not or if it does not matter.

I am reading through PowerBI (PBI) and through Azure data factory (ADF) one folder in a container of a storage account. In that folder there are many files, but also some subfolders with also many files. When I want to read only the files in this folder, but not the subfolder, ADF reads everything by default regardless of whether it is in the folder or in subfolders. In PBI I was able to not read the files in the subfolders but in ADF I have not been able yet...

Should we change this way of storing data or this is not something that really matters? What can be consdiered as best practices? How can I with ADF read only the folder files but not the subfolders?

Any help is welcome, Thx

Answer 1

To avoid copying files from subfolders, you need to uncheck the Recursively option in the copy activity source.

Sample demo:

source container:

Give the path till the required folder. Here, I have given till the source container.

Give the target location as well.

In the source uncheck the Recursively option.

Execute the pipeline and only files from the required main folder will be copied.

Here, I have used binary dataset but the process is same for any kind of dataset.