How to add new column to stream in Pentaho DI, without blocking the stream, new column value is not constant

Question

I have such pipeline result from main stream in sv node as below

and result of node secondary node which in the top of the picture

and at the result I should get such result

how to get result like result of cartesian but non blocking the stream

criteria:

1. it should not block streaming
2. but it should produce a cartesian join type result
3. implement via ? the sign in the table_input itself will not work either, since in a real case it already works with the delta

How to add new column without blocking the stream?

Answer 1

Since the table_cnt is dynamic and will change record, you could set it as a variable. You transformation will look something as below

Table Input --> Set Variable (total_cnt as variable set at ROOT level or Job Level)

Next, the main transformation can use the Get Variable step to get the rows that was set previous. A sample workflow will look as below.

Table Input --> Get Variable --> ... rest of the flow

This approach doesn't use JOIN ROW step and also doesn't block the stream. It appends rows the existing main table input flow. To know more about using variables in Pentaho, you can refer to the Pentaho wiki.

Answer 2

Rishus sollution will work. A much easier sollution is not using cartesian product, but just add to both flows a column (1::int as key) and then connect with stream lookup on that key, and then in SV2 dump the key...