In Pentaho Data Integration (V7), we have activated Kettle's own log tables to generate an error reporting system. So far, we have activated the Job, Transformation, and Channel log tables:
- In the jobs, we have activated the Job and Channel logs.
- In the transformations, all three.
We have several chains, and the system is working well in all of them except for the first chain. The first chain, like the others, calls several jobs which in turn call other jobs and transformations.
All these elements are properly saving the logs at the Job and Transformation level, but they are not filling the Channel table.
When I run the job manually (same job calling to a dummy sub job, the original is too large) , everything works fine, but when the original is running in batch from the task scheduler, this table is not being filled.
Even when launching it from the task scheduler (dummy version) in isolation, it works correctly; it is during the nighttime execution that it does not. It seems curious to me that the Channel table is not completed neither for the main job nor for any of the dependent jobs or transformations. From the second chain onwards, everything works correctly. I have tried regenerating the main job, but it has not worked.
I've found out the channel table is not filled until the job has finished. I am wondering if, due to the job is really large and it takes around 10 horus in finishing , there exist some "timeout" that makes logs not get stored if the process hasn't finished after X time.. i am looking for documentation but I am not finding nothing detailed about the function of these log tables.
In case it helps anyone, in our case, the problem was solved by increasing the value of the KETTLE_MAX_LOGGING_REGISTRY_SIZE and KETTLE_MAX_JOB_ENTRIES_LOGGED variables