Bash: surprising behavior of `fc-list | grep -q` with `pipefail`

Question

I'm losing my sanity here...

Suppose that I want to echo the word "Sans", and then check with grep whether it contains the substring "Sans" or "CrazyStuff":

#!/bin/bash

echo Sans | grep -q Sans       && echo 'y ' || echo ' n'
echo Sans | grep -q CrazyStuff && echo 'y ' || echo ' n'

set -o pipefail

echo Sans | grep -q Sans       && echo 'y ' || echo ' n'
echo Sans | grep -q CrazyStuff && echo 'y ' || echo ' n'

As expected, "Sans" contains "Sans", and does not contain "CrazyStuff", so regardless of the pipefail settings,

• the 1st grep Sans succeeds
• the 2nd grep CrazyStuff fails
• the 3rd grep Sans succeeds
• the 4th grep CrazyStuff fails

The output is:

y
 n
y
 n

So far so good.

---

Now, let's replace the constant string "Sans" by the output of fc-list (it lists installed fonts; You'll probably also have some Sans-Serif font, so it should contain Sans):

#!/bin/bash

captured="$(fc-list)"

echo "$captured" | grep -q Sans       && echo 'y ' || echo ' n'
echo "$captured" | grep -q CrazyStuff && echo 'y ' || echo ' n'

set -o pipefail

echo "$captured" | grep -q Sans       && echo 'y ' || echo ' n'
echo "$captured" | grep -q CrazyStuff && echo 'y ' || echo ' n'

Since the $captured output contains the substring Sans, this program behaves exactly as the first one, the output is again:

y
 n
y
 n

---

Now, the daredevil stunt: instead of echoing the $captured output of fc-list, we simply invoke fc-list four times:

#!/bin/bash

fc-list | grep -q Sans       && echo 'y ' || echo ' n'
fc-list | grep -q CrazyStuff && echo 'y ' || echo ' n'

set -o pipefail

fc-list | grep -q Sans       && echo 'y ' || echo ' n'
fc-list | grep -q CrazyStuff && echo 'y ' || echo ' n'

Obviously, it must behave in exactly the same way as the previous two examples, so unsurprisingly, the output is

y
 n
 n
 n

...wait, what?

How can replacing the constant $captured output of fc-list by actual invocations of fc-list change anything at all? It's always just returning exactly the same list of fonts every time.

Can someone please explain what is going on here?

Even more importantly: how do I fix it?

Thanks in advance.

Answer 1

The problem (as others have pointed out) is that grep -q exits (closing its end of the pipe) before fc-list has finished writing to its end of the pipe, so fc-list gets a SIGPIPE signal when it tries to continue writing, and exits with an error.

The only reason this doesn't happen with echo is that echo sends its output faster, so it finishes writing before grep has time to find its match and exit. If echo were slower, grep were quicker, or the output large enough (especially, large enough to fill the pipe's buffer, so echo couldn't write everything immediately), you'd see this problem with echo as well. This type of timing dependent behavior is known as a "race condition", and rather than trying to fix the timing, the correct solution is to write code that isn't timing-dependent.

For pipe failures like this, there are several possible solutions:

1. Don't use pipefail. It has an annoying tendency to cause problems like this, especially if you don't fully understand how the programs in a pipe work & interact with each other and the pipe.

2. Avoid using things that don't read their entire input.

   In this case, you could replace grep -q somepattern with grep somepattern >/dev/null; this gives the same status indication grep -q would, but it always reads the entire input (and prints matches, but the redirect discards that).

   Similarly, you could replace head -n30 with sed -n "1,30 p".

3. Force success status for commands in the middle of the pipe, e.g. replacing ... | fc-list | ... with ... | (fc-list || true) | ... (but note that this suppresses all errors from fc-list, not just pipe failures).

Answer 2

I suspect grep will quit as soon as it finds a match. When it quits, it breaks the pipe: echo has not finished sending everything. echo is terminated abnormally due to a broken pipe signal (SIGPIPE), and so fails.

You have set pipefail, so that makes the whole pipeline be considered failed.

This is downside of pipefail. When you have command | grep -q pattern, you usually want that to fail if, and only if, the pattern did not occur. If command fails, that doesn't matter; if it produced a match before the failure, then that was satisfied.

The POSIX shell, and its predecessors, was designed around this type of use case; Bash introduced pipefail for those situations in which you need to check for success from all the pipeline elements. It won't work if any of the second or subsequent pipeline elements abruptly quit, causing the pipe to break for the previous ones.