Matillion Grid Iterator Running Random Job order

Question

I have SQL in a "Query Result to Grid" Component similar to the following: -

SELECT JOB, BATCH_ID
FROM   ELT_JOBS
ORDER BY JOB, BATCH_ID ASC

The values are saved in a Grid Variable called "BATCHES" and despite the "ORDER BY" being applied correctly, when the jobs run it's in a random order: I'd expect batches 1 to 10 to run in sequence but instead they're run in a chaotic order like 5,3,6,10,1 etc.

To correct this I've added a Python Component between the "Query Result to Grid" component and the "Grid Iterator" component that uses the following code: -

l_BATCH_IDS = context.getGridVariable('BATCH_IDS')

l_BATCH_IDS.sort()

context.updateGridVariable('BATCH_IDS',l_BATCH_IDS)

which fixes the problem, but it shouldn't be occurring in the first place.

Has anyone else encountered this issue and can you suggest what the root problem is please?

Answer 1

According to documentation, "In Sequential mode, the iterations will always proceed in the same order that the data is held in the Grid Variable."

If you are running in concurrent mode, that may be the issue. Otherwise, I suspect this is a bug in the version of Matillion you are using