Loading only distinct records into table via IICS

Question

I have a 4 flatfiles which are coming everyday,I'm taking these r flatfiles as list files as indirect file load. My requirement: I have to load these records in 4 flatfiles into the database. But the problem is I have to load only distinct records to the target and ignore the duplicate records.here I don't have any primary key ,it has 10 columns ,so all 10 columns should be considered as primary keys Could you please let me know how to load only distinct records to Target table using informatica cloud. Everyday I have to process 4 source files .I need to lookup on target and compare with the source files,and load only the distinct records.i cannot load the record which is already present in target. Please let me know your inputs. Thanks

Answer 1

Use sorter to get distinct right before target. And in the advanced tab, choose to get distinct values like pic below.

Now, if the 4 files has full load you can easily use sorter. But if they have incremental data, you need to do a lookup into the target and use a filter to exclude existing rows.

Depending on size of files, you can have perf problem but you have to increase memory size to handle it properly.