How to see all jobs in a cluster using Spring Batch?

Question

I'm trying to determine all the things I need to consider when deploying jobs to a clustered environment.

I'm not concerned about parallel processing or other scaling things at the moment; I'm more interested in how I make everything act as if it was running on a single server.

So for I've determined that triggering a job should be done via messaging.

The thing that's throwing me for a loop right now is how to utilize something like the Spring Batch Admin UI (even if it's a hand rolled solution) in a clustered deployment. Getting the job information from a JobExplorer seems like one of the keys.

Is Will Schipp's spring-batch-cluster project the answer, or is there a more agreed upon community answer?

Or do I not even need to worry because the JobRepository will be pulling from a shared database?

Or do I need to publish job execution info to a message queue to update the separate Job Repositories?

Are there other things I should be concerned about, like the jobIncrementers?

BTW, if it wasn't clear that I'm a total noob to Spring batch, let it now be known :-)

Answer 1

Spring XD (http://projects.spring.io/spring-xd/) provides a distributed runtime for deploying clusters of containers for batch jobs. It manages the job repository as well as provides way to deploy, start, restart, etc the jobs on the cluster. It addresses fault tolerance (if a node goes down, the job is redeployed for example) as well as many other necessary features that are needed to maintain a clustered Spring Batch environment.

Answer 2

I'm adding the answer that I think we're going to roll with unless someone comments on why it's dumb.

If Spring Batch is configured to use a shared database for all the DAOs that the JobExplorer will use, then running is a cluster isn't much of a concern.

We plan on using Quarts jobs to create JobRequest messages which will be put on a queue. The first server to get to the message will actually kick off the Spring Batch job.

Monitoring running jobs will not be an issue because the JobExplorer gets all of it's information from the database and it doesn't look like it's caching information, so we won't run into cluster issues there either.

So to directly answer the questions...

Is Will Schipp's spring-batch-cluster project the answer, or is there a more agreed upon community answer?

There is some cool stuff in there, but it seems like over-kill when just getting started. I'm not sure if there is "community" agreed upon answer.

Or do I not even need to worry because the JobRepository will be pulling from a shared database?

This seems correct. If using a shared database, all of the nodes in the cluster can read and write all the job information. You just need a way to ensure a timer job isn't getting triggered more than once. Quartz already has a cluster solution.

Or do I need to publish job execution info to a message queue to update the separate Job Repositories?

Again, this shouldn't be needed because the execution info is written to the database.

Are there other things I should be concerned about, like the jobIncrementers?

It doesn't seem like this is a concern. When using the JDBC DAO implementations, it uses a database sequence to increment values.