how to read data from multiple tables in db using spring batch

Question

I tried reading data from one table and writing to other table using spring batch but now my requirement is to read data from mutiple tables and write to a file, so we can achieve this by defining mutiple jobs but I want to do it using single job means single reader and single writer and single processor.

Please provide me some references for this scenario.

Answer 1

Not possible by the classes provided by the spring batch but you can make a way our of it.
Just before the chunk processing add one step, make a custom tasklet where you will assign different sql and different output file and make them run in loop as long as there are sqls to execute.
It might sound difficult but I have worked on same situation, Here is some idea how you can do it -

   <flow id="databaseReadWriteJob">
            <step id="step1_setReaderWriter">
                <tasklet ref="setReaderWriter" />
                <next on="FAILED" to="" />
                <next on="*" to="dataExtractionStep" />
            </step>
            <step id="dataExtractionStep">
                <tasklet>
                    <chunk reader="dbReader" writer="flatFileWriter" commit-interval="${commit-interval}" />
</tasklet>
                <next on="FAILED" to="" />
                <next on="*" to="step3_removeProcessedSql" />
            </step>
            <step id="step3_removeProcessedSql">
                <tasklet ref="removeProcessedSql" />
                <next on="NOOP" to="step1_setReaderWriter" />
                <next on="*" to="step4_validateNumOfSuccessfulSteps" />
            </step>
    </flow>

and here is the bean for setReaderWriter

<beans:bean id="setReaderWriter" class="SetReaderWriter">
        <beans:property name="reader" ref="dbReader" />
        <beans:property name="flatFileWriter" ref="flatFileWriter" />
        <beans:property name="fileSqlMap" ref="jobSqlFileMap" />
        <beans:property name="fileNameBuilder" ref="localFileNameBuilder" />
        <beans:property name="sourceFolder" value="${dataDir}" />
        <beans:property name="dateDiff" value="${dateDiff}" />

Anything you need to add dynamically in Reader or Writer. Above sqlMap is the map of sql as key and Output file as value of that.
I hope it could help.

Answer 2

You may try the things detailed in this answer. In short, you may use CompositeItemReader adding individual readers to it to read from whatever db you want. Each reader can have its own datasource so you can actually read from different servers altogether. Each reader can also have its own query so you can read from different tables also. Then in the unifying row mapper, combine the two tables into one for further processing. And if you need to read those tables from the same DB, you may use joins etc if possible in the query itself.