I am getting below Error while running my pipeline. Pipeline takes files from HDFS, Merges them and again stores the files on HDFS.
Error:
WARN Ignoring file 'Filename and Location' in spool directory as is lesser than offset file. Category : SpoolDirRunnable.
Kindly let me know the possible fix.
This file probably was already processed by this pipeline. You can try to run the pipeline with "reset origin" option.