Python subprocess hangs with psql command

Question

I am running the following piece of python code (runs a command in a shell and grabs its output or reports an error)

import sys
import subprocess
def check_output(args, communicate=None, quiet=False, **kwargs):
    for stream in ["stdout", "stderr"]:
        kwargs.setdefault(stream, subprocess.PIPE)

    proc = subprocess.Popen(args, **kwargs)
    try:
        out, err = proc.communicate()
    finally:
        for f in (proc.stdout, proc.stderr):
            if f is not None:
                f.close()
        proc.wait()

    if kwargs["stderr"] != subprocess.PIPE:
        err = ""

    if proc.returncode != 0:
        raise Exception(args, proc.returncode, err)
    else:
        if not quiet:
            sys.stderr.write(err)
            sys.stderr.flush()
    return out

with the following arguments:

env = dict(
        PGHOST='{pg_host}',
        PGPORT='{pg_port}',
        PGDATABASE='{pg_dbname}',
        PGUSER='{pg_user}',
        PGPASSWORD='{pg_password}',
      )

cmd = ['psql', '-c', "INSERT INTO {ft_geom} SELECT * FROM {ft_geom_in};"].format(**tables)
check_output(cmd, shell=True, env=env)

Here env simply contains the PG[HOST|USER|DATABASE|PORT|..] environment variables and tables contains just the names of those 2 tables. When I run this code, it hangs indefinitely on proc = subprocess.Popen call. I am using python 2.6.5 on Ubuntu 10.04.3 LTS

I check that no tables are locked with the following:

SELECT a.datname,
         c.relname,
         l.transactionid,
         l.mode,
         l.granted,
         a.usename,
         a.current_query, 
         a.query_start,
         age(now(), a.query_start) AS "age", 
         a.procpid 
    FROM  pg_stat_activity a
     JOIN pg_locks         l ON l.pid = a.procpid
     JOIN pg_class         c ON c.oid = l.relation
    ORDER BY a.query_start;

And it shows that all the locks were granted. Not sure where else to look at. I need shell=True because the commands are sometimes more complex, requiring bash pipes. I know I should ideally pass stdout.PIPE of one command to the other, but its not possible to change this at the moment.

Running the very same command from bash directly works as expected, also, running without shell=True works

Answer 1

The problem was that I didn't filter newline characters in tokens passed to check_output, then shell would hang waiting for more input. So make sure you escape the tokens when you pass them to shell.