We have the below requirements in the Nifi flow
a) Fetch a file from HDFS and transfer it to a Linux Server. I have achieved this using FetchHDFS and PutSFTP NiFi Processors, and it is working fine as expected for a single file.
Now, we have a requirement that this HDFS directory may contain multiple files, and all of them need to be transferred to the Linux server. I tried passing the HDFS directory path in the "HDFS File Name" Property of the FetchHDFS processor; however, it seems to accept only file names. Could you please suggest an alternative processor to achieve this?
I tried to use ListHDFS -> FetchHDFS -> PutSFTP Actually, the HDFS files are output of sqoop jobs. I tried connecting RouterAtribute to ListHDFS, but that connection is not allowed.
b) The second requirement is to combine all the files in a specific HDFS directory. Can we achieve this using the MergeContent processor?
Could you please provide some inputs?