I have a parameterized ADF pipeline that is feeded data from a metadata table that is stored in SQL db. One of the columns in the metadata table is 'IsActiveFlag.' Whilst developing the pipeline I used this parameter as condition to filter out records in the metadata table where IsActiveFlag = N.
As I am preparing to deploy to production I am trying to find a way I can improve my pipeline so that I won't be dependent on the IsActiveFlag filter. Ideally I would be able to update the IsActiveFlag some other dynamic way instead of having to change the values manually and then deploy to production based on user demand.
Does anyone have any tips or recommendations on how I can improve my pipeline?
Thanks in advance!
Having 'IsActiveFlag' is a good way to filter out the tablenames you don't want to ingest. If you would perform delete the table entries which you don't want to ingest in , it would be kind of hard delete which is not recommended. Better to go with IsActiveFlag .