SOQL Select all columns in Azure Data Factory (Incremental Load from Salesforce)

1.5k Views Asked by At

Azure Data Factory gives an option to load data incrementally by using an SOQL query, Example bellow:

Select COLUMN_1,...,COLUMN_N from Account Where COLUMN_X = 'VALUES_X'

There is another solution: Delta copy from a database with a control table; but it is dedicated to Azure SQL Database and doesn't take into consideration other Data Sources like Salesforce: https://learn.microsoft.com/en-us/azure/data-factory/solution-template-delta-copy-with-control-table

Thus, we always have to use an SOQL query. The problem with it, is that we can't do a SELECT ALL as in SQL, example bellow:

Select * from Account

So we have to write manually tens of columns for multiple tables!

Is there any way to launch an SOQL query from a Copy Data Activity in Azure Data Factory without mentioning all the columns?

Best regards,

1

There are 1 best solutions below

1
On

A SOQL equivalent of SELECT * (SELECT FIELDS(ALL)) will become available in the Spring '21 release.

Salesforce Object Query Language (SOQL) now makes it easy to include pre-defined groupings of fields within a query statement using the new FIELDS() function.

How: Use FIELDS(ALL), FIELDS(STANDARD), or FIELDS(CUSTOM) in your SELECT statements. For more information, see FIELDS() in the SOQL and SOSL Reference.

Until Spring '21 is released, using hard-coded SOQL queries or dynamically generating them by scripting access to the org's Describe API are the only routes.