Handle file input/output of external binary from python

Question

From my Python script I need to call consequently two external binaries in order to process file in two steps:

import os, subprocess
sbp = subprocess.Popen(['program_1', '-i', 'input.file', '-o', 'temp.file'])
sbp = subprocess.Popen(['program_2', '-i', 'temp.file', '-o', 'output.file'])                      
os.remove('temp.file')

However, it would be nice to speed-up the pipe and reduce disk usage by using virtual RAM-based files instead of 'physical' disk-based. I know that I can use StringIO or tempfile.SpooledTemporaryFile() to handle virtual files within Python script, but is there a possibility to pass the link to such file to an external binary?

Answer 1

from subprocess import Popen
from tempfile import NamedTemporaryFile

tmp = NamedTemporaryFile('w+')
sbp = Popen(['program_1', '-i', 'input.file', '-o', tmp.name])
sbp = Popen(['program_2', '-i', tmp.name, '-o', 'output.file'])                      
tmp.close()

At the end tmp will be delete.

Answer 2

Assuming that you can tell your 2 programs to read and write to/from stdin and stdout, you can just pipe from one subprocess command to the other:

import os, subprocess
sp1 = subprocess.Popen(['program_1', '-i', 'input.file'], stdout=subprocess.PIPE)
sp2 = subprocess.Popen(['program_2', '-o', 'output.file'], stdin=sp1.stdout)
sp1.stdout.close()
sp2.communicate()

See https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline.

Another option (Unix) is to use a named pipe (created at the OS level, e.g. mkfifo /tmp/mypipe):

import os, subprocess
os.mkfifo('/tmp/mypipe')
sp1 = subprocess.Popen(['program_1', '-i', 'input.file', '-o', '/tmp/mypipe'])
sp2 = subprocess.Popen(['program_2', '-i', '/tmp/mypipe', '-o', 'output.file'])

And it should also be possible to use os.pipe().