Issue using subprocess to run a PDAL bash command from Python

Question

Issue:

I cannot run a pdal bash command from Python using subprocess.

Here is the code

based on Running Bash commands in Python:

import os, subprocess

input = '/path/to/file.ply'
output = '/path/to/statfile.json'
if not os.path.isfile(output):
    open(output, 'a').close()

bashcmd = ("pdal info --boundary "
           +input
           +" > "
           +output
           )

print("Bash command is:\n{}\n".format(bashcmd))

process = subprocess.Popen(bashcommand.split(),
                           stdout=subprocess.PIPE,
                           shell=True)
    output, error = process.communicate()
    print("Output:\n{}\n".format(output))
    print("Error:\n{}\n".format(error))

Which gives me this output in the Python console:

Bash command is:
pdal info --boundary /path/to/file.ply > /path/to/statfile.json

Output:
Usage:
  pdal <options>
  pdal <command> <command options>
  --command        The PDAL command
  --debug          Sets the output level to 3 (option deprecated)
  --verbose, -v    Sets the output level (0-8)
  --drivers        List available drivers
  --help, -h       Display help text
  --list-commands  List available commands
  --version        Show program version
  --options        Show options for specified driver (or 'all')
  --log            Log filename (accepts stderr, stdout, stdlog, devnull as
      special cases)
  --logtiming      Turn on timing for log messages

The following commands are available:
  - delta
  - diff
  - fauxplugin
  - ground
  - hausdorff
  - info
  - merge
  - pcl
  - pipeline
  - random
  - smooth
  - sort
  - split
  - tindex
  - translate

See http://pdal.io/apps/ for more detail

Error:
None

It looks as if it had stop reading the arguments of the command after the call to 'pdal' only, which prints this help message.

If I copy the output of the first print and paste it in a bash terminal, it works properly, giving me the output file with the desired metadata. But from Python no output file is created.

Question:

I wonder why (e.g. is there anything wrong with the redirection or the fact that the computation itself may take ~20sec normally?), and how to execute this command from Python?

This doesn't provide a clear enough answer to the present issue.

Answer 1

There are multiple errors here.

• You are using an undefined variable bashCommand instead of the one you defined above bashcmd.
• You are mixing output to a Python file handle with shell redirection.
• You are not capturing the stderr of the process. (I will vaguely assume you do not need the standard error anyway.)
• You should not split() the command if you run it with shell=True.

More broadly, you should probably avoid the shell=True and let Python take care of the redirection for you by connecting the output to the file you open; and in modern times, you really should not use subprocess.Popen() if you can use subprocess.run() or subprocess.check_call() or friends.

import subprocess

input = '/path/to/file.ply'
output = '/path/to/statfile.json'

with open(output, 'a') as handle:
    bashcmd = ['pdal', 'info', '--boundary', input]

    #print("Bash command is:\n{}\n".format(bashcmd))

    result = subprocess.run(bashcmd, stdout=handle, stderr=subprocess.PIPE)

    # No can do because output goes straight to the file now
    ##print("Output:\n{}\n".format(output))
    #print("Error:\n{}\n".format(result.stdout))