Build system that allows sharing modules amongst different binaries

Question

I'm trying to choose the most appropriate build system to work in enterprise with a common source repository, emphasizing sharing of common code. I'd like the source hierarchy to look something like this:

 - src
   - java
     - common
       - net
       - database
     - team1
     - team2
     - team3
       - lib
 - tests
   - java
     - common
       - net
       - database
     - team1
     - team2
     - team3
       - lib

The goal is to have a build system where team[1-3] can have independent builds that explicitly specify their dependencies. Dependencies might look like:

 - team1
   - common/net
   - team3/lib
 - team2
   - common/database
 - team3

So, for example, the build for team1 would include everything within the team1, common/net, and team3/lib; but nothing else. Ideally, tests would be integrated in the same fashion (testing team1 would run tests for team1, common/net, and team3/lib).

I'm currently using Ant, but haven't found a sane way to manage a hierarchy like this. I started to look at Maven 2 for its ability to manage a dependency hierarchy, but it seems to want full-fledged projects for each module. That wouldn't be a problem, but it seems to force me into a directory structure that does not map well to the traditional java package hierarchy. It seems like I might be able to do what I want with buildr using an alternative layout, but I'm worried that might prove to be brittle.

Can someone recommend something that might work for me?

Answer 1

I started to look at Maven 2 for its ability to manage a dependency hierarchy, but it seems to want full-fledged projects for each module.

That's one way to do it. Alternatively, a multi-module Maven project can be organized like this:

project
    module-1
        src
            main
                ....
            test
                ....
        pom.xml
    module-2
        src
            main
                ....
            test
                ....
        pom.xml
    ...
    pom.xml

where each pom.xml could also refer to modules defined by other trees. BTW, the Eclipse maven plugin supports this approach as well as the more common one-module-per-project approach.

Answer 2

What you are going for is going to give you lots of trouble in the long-term... each standalone component should really be made into its own project with its own repository, otherwise, you can get into lots of issues with changes in one component breaking the other components and updating taking excessively long. I strongly recommend that you make each component into its own project and using Maven2 to build.

Answer 3

As you say, Maven 2 is the preferred option for your case. Maven folder structure is not madnatory - it is configurable, if you consider it unsiutable. However, I think it is a good structure that you can follow without remorse.

You can use a repository manager so that people who use some dependencies don't necessarily need to checkout the projects they depend on.

Answer 4

I'm currently using Ant, but haven't found a sane way to manage a hierarchy like this.

This is surprising as Ant (+Ivy?) gives you all the flexibility you want.

I started to look at Maven 2 for its ability to manage a dependency hierarchy, but it seems to want full-fledged projects for each module.

If by this you mean one pom.xml per module, then that's correct.

That wouldn't be a problem, but it seems to force me into a directory structure that does not map well to the traditional java package hierarchy.

Yes, Maven comes with some conventions, the project directory structure being one of them. This is (a bit) configurable though but I don't think you'll be able to match the wanted layout (with tests and sources into separated hierarchies). And actually, I would strongly advice to use defaults if you go for Maven, you should adopt its philosophy, it will save you a lot, really a lot, of pain (not even mentioning that some plugins might use these default in an hard coded way).

To be honest, I don't really understand what you mean by a directory structure that does not map well to the traditional java package hierarchy. First, Maven is perfect for Java, so this doesn't make any sense to me. Second, and this might be more subjective, your layout (with separated tests and sources trees) doesn't look traditional at all to me. Maybe you should clarify what you mean exactly by traditional...

It seems like I might be able to do what I want with buildr using an alternative layout, but I'm worried that might prove to be brittle

I don't know buildr really well so I can't say much about it but I know it is indeed more flexible. That said, if Ant doesn't give you satisfaction in terms of flexibility, then I don't see why buildr would be better.

And don't forget that buildr and Ant+Ivy have much smaller communities compared to Maven. Don't underestimate this, this might become a real concern.

Personally, I would go for Maven and reconsider your layout. But let's say I'm biased.

Answer 5

I think you actually have three issues here.

• How to layout your project so that the artifacts make sense.
• How to best handle the sharing of these artifacts for each project.
• How to handle the loss in productivity while converting the development team to use the new project structure.

For the first issue, try to use Maven conventions wherever possible and organize the project into multiple artifacts. If the artifacts should be nested under a parent, do so. Start off with the simplest artifact which has no dependencies and work your way through the code.

I'm not sure why you believe the layout won't support the traditional Java hierarchy? It should work, especially if you use parent poms.

Obviously the second issue can become quite a handful depending upon how you handle the first one. I would err on the side of creating more artifacts instead of fewer and using a repository manager like Nexus or Artifactory to manage them. At least that way, your team's builds can rely on pre-built and tested jars by hitting your repository to pull down the latest SNAPSHOT or RELEASE of the jar they are working with.

For the third, make sure you're using IDEs that have Maven support. If you're stuck using something like Rational Application Developer 7.0.x or an IDE based on something less than Eclipse 3.4, then you won't be able to use the M2Eclipse plugin. Without M2Eclipse, the developers will have to jump through some manual hoops which are not ideal. Netbeans 6.7 and 6.8 have very good Maven support.

Answer 6

You can do it with Buildr. You could live for some time with it.

Of course, like most people on the thread, I would rather not recommend this approach. You can also use base_dir to change the base directory of the projects.