Copy files of different formats in different folders based using Azure Data Factory

I am new to Azure Data Factory and I am trying to solve a particular use case. I have to copy files from source folder to target folder both of which are in the same storage account. The files in the source folder are of different format (csv, txt, xml) and have date appended at the end, eg: addresses_2020-11-01.csv (date format: yyyy-mm-dd)

I have to create a pipeline that will sort and store files in the dynamic folders in this hierarchy: ex: csv->yyyy->mm->dd. My understanding is first I have to filter the files into different formats and then use split function to split the substring where there is _ and then dynamically create the folder based on the year, month, and day in the filename. Below is the screenshot of the pipeline that I have created so far: I am not able to display the screenshot but the link opens the screenshot.

[Pipeline to filter files, and copy to the destination folder]

What I have done:

1. Use Get Metadata to extract childitems
2. Filter the output from Get Metadata into csv, txt, and xml files
3. Use For each activity that contains a Copy activity. This activity copies files from filter activity into respective folders (csv, txt..) since the wildcard contains *.txt, *.csv, *.xml

I am not sure what is the correct way to move forward once the files are filtered so that dynamic folders are created based on the dates in the filename. I think I need to use set Variable activity along with copy activity but not sure how to accomplish this. Any help will be appreciated.

Thanks!!

https://stackoverflow.com/questions/66680167/copy-files-of-different-formats-in-different-folders-based-using-azure-data-fact March 18, 2021 at 03:43AM