Azure Data Factory gives an option to load data incrementally by using an SOQL query, Example bellow:
Select COLUMN_1,...,COLUMN_N from Account Where COLUMN_X = 'VALUES_X'
There is another solution: Delta copy from a database with a control table; but it is dedicated to Azure SQL Database and doesn't take into consideration other Data Sources like Salesforce: https://learn.microsoft.com/en-us/azure/data-factory/solution-template-delta-copy-with-control-table
Thus, we always have to use an SOQL query. The problem with it, is that we can't do a SELECT ALL as in SQL, example bellow:
Select * from Account
So we have to write manually tens of columns for multiple tables!
Is there any way to launch an SOQL query from a Copy Data Activity in Azure Data Factory without mentioning all the columns?
Best regards,
A SOQL equivalent of
SELECT *
(SELECT FIELDS(ALL)
) will become available in the Spring '21 release.Until Spring '21 is released, using hard-coded SOQL queries or dynamically generating them by scripting access to the org's Describe API are the only routes.