How can I fetch a dynamic file from FTP server-generated every day?

Question

There are transactional input CSV files coming on a daily basis on an FTP location. I need to read these input files and process them on daily batch execution. The name of the files remains the same every day, but the date gets appended at the end of the filenames every day,
Ex:

Day1
    General_Ledger1_2020-07-01,
    General_Ledger2_2020-07-01,
    General_Ledger3_2020-07-01,
    General_Ledger4_2020-07-01,
    General_Ledger5_2020-07-01

    Day2
    General_Ledger1_2020-07-02,
    General_Ledger2_2020-07-02,
    General_Ledger3_2020-07-02,
    General_Ledger4_2020-07-02,
    General_Ledger5_2020-07-02

How can I append this Date information to the input file name every time the job runs?

Answer 1

I have faced similar problem earlier and this can be solved using calculated parameter in the file path. Here, you can create expressions that will retrieve the file dynamically.

Example,

 CONCAT( UPPER(lit('$(Prefix)')), ADD_DAYS( TODATE(lit('$(currentTime)'), 'yyyy-mm-dd'), 'yyyy-mm-dd' ,-1),'.csv')

Breaking of the expression :

• $(currentTime) : this system parameter will get the current date (this will also include timestamp).

• (TODATE(lit('$(currentTime)'), 'yyyy-mm-dd') : TODATE will get only date from the whole timestamp with format as ‘yyyy-mm-dd’.

• ADD_DAYS(TODATE(lit('$(currentTime)'), 'yyyy-mm-dd'), 'yyyy-mm-dd' ,-1) : ADD_DAYS here will add -1 to the date retrieved from. TODATE(). Hence (2020-04-24) + (-1) would give us 2020-04-23

• $(Prefix) : $(Prefix) will be an user defined input parameter of type String which user will be providing at runtime – Since the prefix will be always dynamic.

• CONCAT() : Finally to combine all the results and form the exact file path CONCAT() can be used. Also in between some static string is added as it will always be fixed for every file to be read.