How to rightly configure an Azure data factory data flow with a delta file as sink?

199 Views Asked by At

I am trying to sink data to a delta file. I want to use a insert and update method using the unique Id column: (sink settings)

enter image description here

Whenever the update method is allowed a AlterRow operation will appear. I have the following settings for the alterRow:

enter image description here

I want a row to be updated whenever the id value is the same (set to true()). I want a row to be inserted whenever the id value is not yet in the delta file (set to false().

Is this correct?

2

There are 2 best solutions below

4
B.Griffin On BEST ANSWER

If you want inserting when the id doesn't exist and updating when it does, you could just select upsert as your update method.

An upsert is a combination of an insert and an update.

The way you've currently got it configured means that it will always update (as it's set to true) and will never insert (as it's set to false).

0
Pratik Lad On

I want a row to be updated whenever the id value is the same (set to true()). I want a row to be inserted whenever the id value is not yet in the delta file (set to false().

To achieve this, you can use the upsert option directly with alter row transformation as below

enter image description here

then you need to set update method as upsert in sink with the key column based on which it will update the destination.

enter image description here