My question has two parts:
- I'm trying to create different queries from a reference table. more or less like this question here.
Now I'm managing to get the output like this from my reference table:
SELECT
A.FactGeneralLedgerId, A.DimLocalCurrencyId, A.DimLocalCurrencyId AS ColumnNames
FROM `devuser_030_serve`.`dbt_finance`.`FactGeneralLedger` A
LEFT JOIN `devuser_020_prepare`.`dbt_to_prep_01`.`ps5_Currency` B
ON A.DimLocalCurrencyId = B.DimCurrencyId
WHERE B.DimCurrencyId IS NULL AND A.DimLocalCurrencyId <> 'MXP'
UNION ALL
SELECT
A.dimcompanyId, A.dimsystemcompanyId, A.SourceSystemId AS ColumnNames
FROM `devuser_030_serve`.`dbt_finance`.`FactGeneralLedger` A
LEFT JOIN `devuser_030_serve`.`dbt_finance`.`DimCompany` B
ON A.Dimcompanyid= B.Dimcompanyid
WHERE (B.Dimcompanyid is null or B.DimCompanyId = '_N/A') AND A.SourceSystemId = 'PE2'
But as this can potentially be a lot of queries the UNION ALL will be a very heavy query to execute. So I would like to use the run_query option from dbt and execute the queries one by one. The result should be stored and being written to a table.
How can I manage this ?
- As my queries will be changed/added I cannot add this part to my models file every time.
-- depends_on: {{ ref('FactGeneralLedger') }}
-- depends_on: {{ ref('ps5_Currency_01') }}
-- depends_on: {{ ref('DimCompany') }}
Is there a way to get this part also generic according to the generated query above ?
This is my full code :
-- depends_on: {{ ref('FactGeneralLedger') }}
-- depends_on: {{ ref('ps5_Currency_01') }}
-- depends_on: {{ ref('DimCompany') }}
{% set query %}
select
OriginSourceTableName
, ColumnNames
, ReferenceSourceTableName
, ReferenceColumns
, WhereClause
, JoinClause
from {{ ref( 'mdm_Screening_01' ) }}
{% endset %}
{% for i in query_to_list(query) %}
{{ log(dbt_utils.pretty_log_format(i), info=True) }}
SELECT
{{ i[1] }} AS ColumnNames
FROM {{ ref([i[0]]|join('')) }} A
LEFT JOIN {{ ref([i[2]]|join('')) }} B
ON {{ i[5] }}
{% if i[4] %}
where {{ i[4] }}
{% endif %}
{% if not loop.last %}
UNION ALL
{% endif %}
{% endfor %}
Thanks a lot in advance !!!