python subprocess does not capture stderr of a pv command

Question

I need to run some bash script with python and show the output. Some of these scripts contains some pv command but from subprocess I am not able to get the pv output.

import subprocess as sp

p = sp.Popen(["./script.sh"], shell=False, bufsize=1, stdout=sp.PIPE, stderr=sp.STDOUT, universal_newlines=True)

p.wait()

print(p.returncode)
print(p.stdout.read())

#!/bin/bash

set -e

echo "aaa"
echo "bbb" >&2

pv -L 1000 -F "%t %b %r %e" /path/to/some/file | cat > /tmp/foo

Runnig this python script I simply get echo's outputs:

$ python script.py 
0
aaa
bbb

Answer 1

pv is sensitive to whether it is running in a terminal. If you want to capture its output, use the pv -f option to force it to display the progress indicator even when it's not connected to a terminal.

For more complex scenarios, you might have to use something like pty to simulate running under a terminal.

(Of course, you might be better off using a native Python progress bar like tqdm.)