Use smart builders or a lot of schedulers to compile and test 5 builds

Question

I want to setup a continuous integration environment with three operating systems (MacOSX, Windows and Linux). I need to build five different builds: win32bit, win64bit, lin32bit, lin64bit, and mac. For each build I need to make the following steps:

-compile
-put some files in the binary folder
-make a 7z-archive of the binary folder
-upload the 7z-archive to a server

Of course there are dependencies. For example it is useless to create and upload a 7z-archive when compiling failed.

My first attempt was to build a small hierarchical system of schedulers and builders, but I don't know how to handle the dependencies in each build:

My plan (Start_scheduler is listening for svn commits):

\Start_scheduler, kicks off win_builder, lin_builder, and mac_builder
 \win_builder, compiles and uploads win32bit and win64bit
   -compile 32bit
   -put some files in the binary folder
   -make a 7z-archive of the binary folder
   -trigger upload_scheduler to upload IF 32bit compile was succesful
   -compile 64bit
   -put some files in the binary folder
   -make a 7z-archive of the binary folder
   -trigger upload_scheduler to upload IF 64bit compile was succesful
 \lin_builder, compiles and uploads lin32bit and lin64bit
   -compile 32bit
   -put some files in the binary folder
   -make a 7z-archive of the binary folder
   -trigger upload_scheduler to upload IF 32bit compile was succesful
   -compile 64bit
   -put some files in the binary folder
   -make a 7z-archive of the binary folder
   -trigger upload_scheduler to upload IF 64bit compile was succesful
 \mac_builder, compiles and uploads mac
   -compile
   -put some files in the binary folder
   -make a 7z-archive of the binary folder
   -triggerupload_scheduler to  upload IF compile was succesful
\upload_scheduler
 -upload 7z-archive to central server
 -send notification about new archive

Basically I have two problems. First, how can I define the IF-dependency between compile and upload, but at the same time make the 64bit compile independent from the 32bit compile: the build system should try to compile 64bit even if 32bit failed. And second, is it possible to parametrize the upload_scheduler so I can reuse it for every build? It would be tedious if I would need to maintain a seperate upload_scheduler for each build.

Answer 1

This kind of thing is handled nicely by Ant and Cruise Control on the Java side.

I wonder if PyAnt could help you with Python.

Answer 2

You pretty much can't beat Hudson (now called Jenkins) as a Continuous Integration system for Python. Google for 'hudson python'.

Answer 3

Instead of using single builder for both 32bit and 64 bit builds you may use separate platform builds which will be running on a single buildslave. In this case 32bit builds are built separately from 64bit builds and one build will never fail another.

Concerning upload scheduler, you might want to look at TriggerableScheduler.