How is restic outputting data to the screen but not to stdout or stderr?

Question

Updating line

I have a question about where the output of a certain command is going. I'm using restic as an example of a command that behaves this way. It's the last line of the command that's of interest to me, because it updates while the restore process is working.

For context, restic's a back-up utility; the command below is restoring the contents of my-repo from a backup stored on example.com back to my local machine.

$ restic restore latest --repo sftp:[email protected]:my-repo --target . --verbose
repository ae412afe opened (version 1)
restoring <Snapshot dff2f51a> to /Users/matt
[0:11] 0.02%  192 files 37.687 MiB, total 39644 files 171.389 GiB

The last line updates over time until it reaches 100%, e.g.

[12:31] 1.22%  2989 files 2.087 GiB, total 39644 files 171.389 GiB

---

Execute from python

I want to run the restore command from a Python script (using subprocess.Popen) and want the script to print out the updating line but haven't been able to get that working.

I suspected it could be that the updating line was being output to stderr instead of stdout. So I repeated the restore command and redirected the stdout and stderr to files, as below:

restic restore lat...rbose > out.log 2> err.log

err.log remains empty, and out.log only contains the restoring <Snapshot dff2f51a> to /Users/matt line, nothing else.

---

Python snippet

Here's the snippet of my code calling Popen:

backup_command = [
    # fmt: off
    "restic",
    "--repo", repo_name,
    "backup",
    "--verbose",
    "--exclude", "node_modules",
    "--exclude", "whatever",
    # fmt: on
]
backup_process = subprocess.Popen(backup_command, env=env_vars)
backup_process.wait()

In reference to jurez's answer below, I'm not passing stdin or stderr, but having looked at the source for Popen, they default to None, as per jurez's suggestion.

---

Questions

I have a couple of questions.

• Why isn't the repository ae412afe opened (version 1) line in either of my output files?
• Where is the [0:11] 0.02% 192 fil... line going? And how can I show it?

Answer 1

From the restic documentation:

When running from a non-interactive console progress reporting is disabled by default to not fill your logs. For interactive and non-interactive consoles the environment variable RESTIC_PROGRESS_FPS can be used to control the frequency of progress reporting. Use for example 0.016666 to only update the progress once per minute.

My use of Popen creates a non-interactive shell, resulting in no progress reporting output. Setting the value works as advertised. I find it easier to think of as "seconds between each update" rathen that "frames per second" so use 1 / INTERVAL_IN_SECONDS to set the FPS.

Credit to jasonharper for looking up the docs, thanks!

Answer 2

From the Popen documentation:

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object with a valid file descriptor, and None. PIPE indicates that a new pipe to the child should be created. DEVNULL indicates that the special file os.devnull will be used. With the default settings of None, no redirection will occur; the child’s file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the applications should be captured into the same file handle as for stdout.

This means that if you use stdin=None and stderr=None in Popen constructor, the subprocess will inherit the STDIN and STDERR file descriptors from Python. In other words, any output that program will send to those streams will appear as Python's own standard output / standard error.