Is it safe to run conda-build in parallel?

Question

First question, please yell at me if I'm not following any established norms here :)

I'm setting up a full conda-build pipeline for the first time and I've got a Makefile that looks something like this:

build:
    echo "Building packages."

    conda-build my_first_package/conda-recipe --output-folder /path/to/dev/channel
    conda-build my_second_package/conda-recipe --output-folder /path/to/dev/channel
    conda-build my_third_package/conda-recipe --output-folder /path/to/dev/channel

    conda index /path/to/dev/channel

The three packages in question are pretty tightly connected to each other and are being stored in the same repo so it would be nice to build each one in the same pipeline. What I'd like to do is spin off a separate process for each of those and run each conda-build command in parallel. Not sure if this is a safe action to take though as I don't really understand what conda-build is doing to "build" the package.

I'll be testing this idea later today and will post an update but what I'm really worried about is that it will work with some unintended side effects and I'm not sure how to go about testing for those. So I think my question is: Can conda-build build multiple packages in parallel into the same conda channel?

Answer 1

Alternatively, for very tightly coupled packages one may want to look into single builds with multiple outputs. For reference, see the Outputs Section documentation of conda-build. It may also be worth checking out a full-blown example, like Conda Forge's matplotlib-feedstock, which builds matplotlib, matplotlib-base, and mpl-sample-data all from a single recipe.

In this particular case, I'd imagine something like

meta.yaml

...
outputs:
  - name: my_first_package
    ...
  - name: my_second_package
    ...
  - name: my_third_package
    ...
...

then run a single conda build command.