Monolith git repo vs micro repos

Question

I have a project with several small batch jobs. Each batch job is quite independent of another (connect to different S3 buckets or REST services using a different file format/REST structure). It will take several months to create all of these batch jobs. Ideally, I would like to release each batch job independently of the others. Especially when it comes to the notion of a production hotfix. It's quicker and easier to release one batch job in an emergency than be forced to re-release all batch jobs (and having to deal with running jobs).

The big hurdle now is git. Should I:

1. Create a new git repo for every batch job
2. Create one monolithic repo for all jobs or
3. Create a repo for some sub-group of batch jobs

I'm leaning toward option 1, but the notion of dozens of git repos seems overwhelming. There is a common repo all jobs will import. If I need to change something in the common repo, I have to do a git dance for every batch job. However, if I stuff everything in a monolithic repo, I either have to have dozens of "master" branches or re-release every job every time.

Is it possible to use micro git repos and use some sort of tool to do simple operations (eg git clone of all repos)? Maybe gradle can somehow do this? Is it better to just use a monolithic repo and have an ugly release process and branching structure?

Answer 1

I feel rather strongly that micro repos are a mixed blessing, so I encourage anybody thinking of heading down that path to read this post first. That said, the mr utility lets you treat an ad hoc collection of repos as a single, logical unit, and it even works with a non-homogeneous collection of repo types (git, svn, hg, bzr, etc). I've found it extremely easy to use—so much so that it has replaced a custom Fabric script I'd been using for a few years to manage multiple repos. If you're only dealing with git repos, Android's repo may be of interest, too.