Data Factory succeeds but no data at target

Question

I am running an incremental process for multiple tables where I am using old watermark, new watermark, copy activity, and a stored procedure to update the watermark value. One of my table succeeds when I run the pipeline, for eg I can see total rows copied as 900 whic matches source SQl but when I check in the Database I can see only 70 rows getting reflected, I am not sure why I am not able to see all the rows at database level. Is this some kind of bug in ADF. Need some advise and suggestions on my situation

Thank You

Answer 1

Its not the first time I am seeing this, its quite common complaint because "rows written" might be confusing to some. Can you implement data consistency verification in copy activity? And what is your fault tolerance?

https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-data-consistency

Some rows may be skipped, in this case for example ADF will still show you like they are "written" because logically they are, but physically not, because of incompatible rows etc. which are skipped. In other words, skipped rows are counted as "written" in ADF terminology. Should check that as well

Answer 2

This happens the rows are rejected at the sink. Do the following to confirm and fix the error.

1. check what setting is enabled under copy activity --> settings --> fault tolerance. if it is skip incompatiable rows, the job won't fail even if the rows are rejected.
2. Under copy activity settings tick enable logging and output the rejected rows to your storage account.
3. The reason for skipping the rows can be identified from this logs.
4. Make necessary changes to the sink table and re run the job.