Nested buildout packages with mr.developer - Recursive buildout

Question

I am using mr.developer to checkout my packages from a mercurial repository, but I must be doing something wrong since I have a problem with nested dependencies.

For example, if I have foo with the following

  [buildout]
  develop = .

  extensions = mr.developer
  sources = sources
  auto-checkout =
      pack1
  parts = foo

  [sources]
  pack1 = hg http://blah.com/hg/pack1

foo has a dependency on pack1, listed in setup.py as install_requires = ['pack1'],

When I run bin/buildout, everything goes smoothly, mr.developer downloads pack1, and foo gets created without issues since pack1 has been downloaded, and therefore exists.

Now, I have another package, bar, which lists foo as a dependency.

  [buildout]
  develop = .

  extensions = mr.developer
  sources = sources
  auto-checkout =
      foo
  parts = bar

  [sources]
  foo = hg http://blah.com/hg/foo

I also list foo as a dependency in my setup.py by doing install_requires = ['foo'], What happens now is the part that I do not understand.

When I run bin/buildout, mr.developer goes and fetch foo, but doesn't seams to execute buildout.cfg located inside of foo/ . As a result, foo/setup.py requires pack1, which doesn't exist.

How to make sure that mr.developer actually goes and fetch pack1 on http://blah.com/hg/pack as indicated in the foo/buildout.cfg?

I would like to be able to nest multiple packages like this, without having to go in depth into each package and run buildout manually.

Cheers, Martin

Answer 1

You are misunderstanding how buildout works.

Normally, buildout will try and find all eggs needed to build your parts for you. It does so by searching for the eggs (optionally pinned to specific versions) in your site-packages, on PyPI or in any additional web locations (using find-links).

It will do so recursively until all dependencies are met. So if you specify you want to use an egg called foo that depends on bar, which in turn depends on spam and bacon, buildout will locate those four eggs for you.

Note that eggs are special python packages, using the .egg extension. If there is a python package with a setup.py file instead, that specifies the correct name, then that setup.py is executed to create an egg on the fly.

This is were development eggs come in; they are python packages that do not need to be downloaded from elsewhere, because they are already present on the filesystem. Their version requirements are not enforced, and if present they take precedence over other versions of the egg found elsewhere. When buildout runs, their setup.py is run to build an egg in-place; you'll find a .egg-info directory in that package when buildout has run and some more metadata is stored in the develop-eggs directory of your buildout.

In your examples you use mr.developer to manage your development eggs, loading them from a mercurial repository first. Buildout itself doesn't really care about this, it's just a (clever) means of loading python packages from a SCM repository and treating them as python eggs.

All you need to do is list all dependencies coming from mercurial in [sources] and in auto-checkout (one per line). In your case the dependencies run bar -> foo -> pack1, and by listing foo and pack1 both in mr.developer-controlled configurations you ensure that buildout will find development eggs for both of these.

In all this it is important to remember that one buildout configuration is all that is needed; buildout does not run buildout configuration files found inside packages. It only deals with python eggs, not other buildout configurations. You do sometimes find buildout configuration files inside python eggs, but these are there for the developer of the egg, to run tests and aid development, not to pull in dependencies when used as an egg in your own projects.