Apache Commons Exec Change PATH and Execute virtualenv's pip

Question

I'm having some difficulty using the Apache Commons Exec library to change the PATH environment variable to point to a created Python virtualenv in my target directory.

Ideally, I want something that is equivalent to activating the Python virtualenv, but in Java. The best way to do this as far as I know is to change environment variables so that its pip and python executables are discovered before my othervenv (which is another virtualenv that I use mainly).

I have this method in my PluginUtils class:

public static String callAndGetOutput(CommandLine commandLine, Map<String, String> environment) throws IOException
    {
        CollectingLogOutputStream outputStream = new CollectingLogOutputStream();
        Executor executor = new DefaultExecutor();
        DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();
        PumpStreamHandler streamHandler = new PumpStreamHandler(outputStream);
        executor.setStreamHandler(streamHandler);
        executor.execute(commandLine, environment, resultHandler);
        try
        {
            // Wait for the subprocess to finish.
            resultHandler.waitFor();
        }
        catch(InterruptedException e)
        {
            throw new IOException(e);
        }
        return outputStream.getOuput();
    }

And then this class that calls this method.

import org.apache.commons.exec.CommandLine;
import org.apache.commons.exec.environment.EnvironmentUtils;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Map;

public class Example
{

    public void run() throws Exception
    {
        Map<String, String> env = EnvironmentUtils.getProcEnvironment();
//        env.forEach((k,v) -> System.out.println(k + "=" + v));
        System.out.println(PluginUtilities.callAndGetOutput(CommandLine.parse("which python"), env));
        System.out.println(PluginUtilities.callAndGetOutput(CommandLine.parse("which pip"), env));
        Path venvDir = Paths.get("", "target", "testvenv");
        Path venvBin = venvDir.resolve("bin");
        assert(Files.isDirectory(venvDir));
        assert(Files.isDirectory(venvBin));
        env.put("PATH", venvBin.toAbsolutePath().toString()+ File.pathSeparator +env.get("PATH"));
        env.put("VIRTUAL_ENV", venvDir.toAbsolutePath().toString());
//        env.forEach((k,v) -> System.out.println(k + "=" + v));
        System.out.println(PluginUtilities.callAndGetOutput(CommandLine.parse("which python"), env));
        System.out.println(PluginUtilities.callAndGetOutput(CommandLine.parse("which pip"), env));
        Path venvPip = venvBin.resolve("pip");
        System.out.println(PluginUtilities.callAndGetOutput(CommandLine.parse("pip install jinja2"), env));
    }

    public static void main(String[] args) throws Exception
    {
        Example example = new Example();
        example.run();
    }
}

The output of this is as follows:

/home/lucas/.virtualenvs/othervenv/bin/python
/home/lucas/.virtualenvs/othervenv/bin/pip
/home/lucas/projects/myproject/mymodule/target/testvenv/bin/python
/home/lucas/projects/myproject/mymodule/target/testvenv/bin/pip
Requirement already satisfied: jinja2 in /home/lucas/.virtualenvs/othervenv/lib/python2.7/site-packages
Requirement already satisfied: MarkupSafe in /home/lucas/.virtualenvs/othervenv/lib/python2.7/site-packages (from jinja2)

I'm confused why which pip would return the correct pip executable while running pip calls the incorrect executable. I was able to use venvPip directly to install jinja2, but I want to avoid passing in absolute paths to pip and instead have it discoverable on the PATH.

I'm thinking that there's possibly a race condition, but I added the DefaultExecuteResultHandler so all the subprocess calls to be synchronous and that doesn't seem to help.

Answer 1

Short answer: One needs to refer to the correct python or pip executable when constructing the command line. One way to make it easy is to store the venv location in a placeholder map e.g.

CommandLine.parse("${VBIN}/pip install jinja2",
    Collections.singletonMap("VBIN", venvBin.toAbsolutePath().toString()))

Technically it should also be possible to launch the command via shell e.g. sh pip install jinja2 but this will not be portable to non-unix systems.

Long answer: The PATH that Java Runtime#exec (that commons.exec ultimately calls on most platforms) uses to search for an executable is not affected by the environment that is later passed to the spawned process.

This is what happens when which pip is launched

1. Runtime#exec consults the PATH passed to the JVM and scans these directories for an executable file called which
2. Runtime#exec finds /usr/bin/which and launches it with a new environment that contains an updated PATH
3. /usr/bin/which consults the PATH passed to it and scans these directories for an executable file called pip
4. Because /usr/bin/which operates off of an updated PATH it finds testvenv/bin/pip and prints its location

This is what happens when pip install jinja2 is launched:

1. Runtime#exec consults the PATH passed to the JVM and scans these directories for an executable file called pip
2. Runtime#exec finds otherenv/bin/pip and launches it with with a new environment that contains an updated PATH
3. otherenv/bin/pip attempts to operate on otherenv and thus fails to perofm the task