Capturing first line from shell command output and continue processing rest

Question

I'm trying to capture and "store away" (preferably in a variable) the first line of a command output and forward the remainder of the output back to stdin of the next command in a pipe. As an FYI, the purpose is to capture and store the cursor reference of a redis-cli SCAN invocation, while processing the data returned.

This is what I came up with:

command | { read first_line ; cat ; } | process_rest
echo $first_line

This seems to work as intended in zsh, but not in Bourne Shell ("/bin/sh") or bash. It appears that piping to the command grouping causes a subshell in sh/bash, "swallowing" the first_line variable.

What is the best way to achieve what I'm trying to do? Will I need to "pipe" the first line through a file or fifo instead of trying to set a variable? I also tried using a while read line ; do construct to filter out the first line and continue processing the rest, but it appears it has the same effect (any variable changes within the do/done block are not reflected in the outer scope).

Answer 1

It's not the braces. In a POSIX shell, any command sequence involving pipes ( cmd1 | cmd2 ) runs both in subshells.

To retain the value of first_line you must not run the read in a subshell.

With bash process substitution you can do:

while
    IFS= read -r first_line
    consumer
do
    break
done < <( producer )

or

{ IFS= read -r first_line; consumer; } < <( producer )

Bash also has a lastpipe option which is available when job control is disabled (typically scripts / non-interactive shells, or when disabled explicitly):

set +m
shopt -s lastpipe

producer | {
    IFS= read -r first_line
    consumer
}

---

BashFAQ 024 has more information and options.

Answer 2

You may consider using a temporary file :

tempfile=$(mktemp)
command | { read -r first_line ; printf "%s\n" "$first_line" > "$tempfile"; cat ; } | process_rest

Answer 3

I think I found a way that works:

first_line=$(command | { read first_line ; echo $first_line ; process_rest })
echo $first_line

This will capture the first line while processing the remainder of the data. The main drawback of this approach is that I cannot easily redirect stdout of process_rest as part of the whole construct anymore. E.g.

get_data() {
  first_line=$(command | { read first_line ; echo $first_line ; process_rest }) 
  echo 2>&1 $first_line
}

get_data | do_something_else # derp

One possible solution to this might be to use a fifo:

get_data() {
  FIFO=$(mktemp -u)
  mkfifo $FIFO
  first_line=$(cat | { read first_line ; echo $first_line ; process_rest > $FIFO})
  echo 2>&1 first_line
  cat $FIFO
  rm $FIFO
}
get_data | do_something_else

Answer 4

If the output of command is textual and fairly small (anything up to 10K lines should be OK on any modern system) then you could store it all in memory and extract the pieces that you want. This is one way to do it:

output=$(command)
readarray -t output_lines <<<"$output"
first_line=${output_lines[0]}
printf '%s\n' "${output_lines[@]:1}" | process_rest

Answer 5

Here is one way:

{ first_line=$(command | {
    IFS= read -r tmp
    printf '%s\n' "$tmp"
    process_rest >&3
  })
} 3>&1