DBT run_query in for loop with macros

232 Views Asked by At

My question has two parts:

  1. I'm trying to create different queries from a reference table. more or less like this question here.

Now I'm managing to get the output like this from my reference table:

SELECT 
    A.FactGeneralLedgerId, A.DimLocalCurrencyId, A.DimLocalCurrencyId AS ColumnNames
FROM `devuser_030_serve`.`dbt_finance`.`FactGeneralLedger` A  

LEFT JOIN `devuser_020_prepare`.`dbt_to_prep_01`.`ps5_Currency` B 
ON A.DimLocalCurrencyId = B.DimCurrencyId 

WHERE B.DimCurrencyId IS NULL AND A.DimLocalCurrencyId <> 'MXP'


UNION ALL 


SELECT 
    A.dimcompanyId, A.dimsystemcompanyId, A.SourceSystemId AS ColumnNames
FROM `devuser_030_serve`.`dbt_finance`.`FactGeneralLedger` A  

LEFT JOIN `devuser_030_serve`.`dbt_finance`.`DimCompany` B 
ON A.Dimcompanyid= B.Dimcompanyid 

WHERE (B.Dimcompanyid is null or B.DimCompanyId = '_N/A') AND A.SourceSystemId = 'PE2'

But as this can potentially be a lot of queries the UNION ALL will be a very heavy query to execute. So I would like to use the run_query option from dbt and execute the queries one by one. The result should be stored and being written to a table.

How can I manage this ?

  1. As my queries will be changed/added I cannot add this part to my models file every time.
-- depends_on: {{ ref('FactGeneralLedger') }}
-- depends_on: {{ ref('ps5_Currency_01') }}
-- depends_on: {{ ref('DimCompany') }}

Is there a way to get this part also generic according to the generated query above ?

This is my full code :

-- depends_on: {{ ref('FactGeneralLedger') }}
-- depends_on: {{ ref('ps5_Currency_01') }}
-- depends_on: {{ ref('DimCompany') }}

{% set query %}
    select 
        OriginSourceTableName
        , ColumnNames
        , ReferenceSourceTableName
        , ReferenceColumns
        , WhereClause
        , JoinClause
    from {{ ref( 'mdm_Screening_01' ) }}
{% endset %}

{% for i in query_to_list(query) %}
{{ log(dbt_utils.pretty_log_format(i), info=True) }}


SELECT 
    {{ i[1] }} AS ColumnNames
FROM {{ ref([i[0]]|join('')) }} A  

LEFT JOIN {{ ref([i[2]]|join('')) }} B 
ON {{ i[5] }} 

{% if i[4] %}
where {{ i[4] }}
{% endif %}

{% if not loop.last %}
UNION ALL 
{% endif %}
{% endfor %}

Thanks a lot in advance !!!

0

There are 0 best solutions below