How to copy good rows from source to destination when we have 1000 rows in Azure Data Factory, in these 1000 rows there are bad rows with several reasons not specific to single reason, So task is we need to copy all good rows by avoiding bad rows.
I have tried by using if-else loop activity to terminate bad rows, but here my concern is how can I now with what reasons they are consider as bad rows. If you have many reasons we need give many else conditions. So please me with automated process to sort out bad rows.
In ADF copy activity, there is a setting to skip incompatible rows:
This would skip them in case if there is any issue w.r.t schema data type etc.
In case if there are rule specific wrong rows, you might have to write your own custom filters via dataflow