I am trying to transfer multiple csv files from the Azure storage container to the GCP bucket through the Data fusion pipeline.
I can successfully able to transfer a single file by mentioning the below path (full path for specific CSV file) for the 'path' field for Azure blob storage configuration wasbs://[email protected]/CSVFile.csv
but, when I am trying to transfer multiple files from 'contrainername' container, below 'path' not working (here I didn't mention any file name after '/' as I need to transfer all the files under this container)
'wasbs://[email protected]/'
It is throwing exception as 'Spark program 'phase-1' failed with error: / is not found. Please check the system logs for more details'
Here I am using 'SAS Token' for authentication which is generated at the container level which is working perfectly for the full path file.
Is there any Forloop option in GCP pipeline to iterate through the files? Thanks in advance
Edit: SAS Token generated at container level not working for the path'/'. SAS Token generated at Storage account level can able to pick all the files under that directory. But it is combining all the files data into a single file(generated at GC storage)
Anyone can help on how to transfer files separately? Thanks