How to fetch files from Azure Blob with current date in the path in Mosaic Decisions

Question

I am using Mosaic Decisions for a specific data pipeline I have to run on a daily basis. I have a client whose file location changes everyday in the blob container based on the current date.

For example, the file sales.xlsx has a path like daily_records/<current-date>/sales.xlsx, where the is a date value in format yyyy-MM-dd.

I have tried to use the System Parameter $(currentTime). However, this parameter also has a timestamp associated with it which doesn't satisfy my use case.

I would like to automatically get the current date when I run the mosaic flow and not have to re-type the date in the path everyday. How can I achieve this in Mosaic Decisions?

Answer 1

I have faced a similar situation before, and the workaround for this is to use calculated parameters rather than system parameters.

First, Create a calculated parameter from the flow config menu with Data Type as Date. Give it a Parameter Name of folderName. Set the Expression value as TO_DATE(TODAY('Asia/Calcutta')). Here, pass in your current Timezone in place of 'Asia/Calcutta' if your current time zone is different from mine. Refer Screenshot below

Then, in the Reader Node configuration menu, add your file path as daily_records/$(folderName)/sales.xlsx. The path would be updated daily with the new date and you can click on validate directly to get the latest file without having to type the new date every day.