I am attempting to execute a program called fasterq-dump. This program takes some folder SRRXXXXXXXX as input and outputs text files. It also allows one to redirect the data for the files it creates straight to standard output with the flag -Z.
I pipe this standard output through awk to stream alternating groups of four lines through pigz to two separate files, 1.fq.gz and 2.fq.gz. Here is the command in its entirety:
fasterq-dump -e 14 -t ./SRR21719548 --split-spot -Z | awk '{if ((NR-1) % 8 < 4) {print | "pigz --fast -p 14 > 1.fq.gz"} else {print | "pigz --fast -p 14 > 2.fq.gz"} }'
Aside from the data it sends to standard output (which I caught and sent to awk), it also prints a few lines after its completion:
spots read : 26,507,360
reads read : 53,014,720
reads written : 53,014,720
I would like to append the above three lines to a text file. I append the 'redirection' additions to my command like so:
fasterq-dump -e 14 -t ./SRR21719548 --split-spot -Z | awk '{if ((NR-1) % 8 < 4) {print | "pigz --fast -p 14 > 1.fq.gz"} else {print | "pigz --fast -p 14 > 2.fq.gz"} }' [REDIRECT_ADDITION]
However, any addition I make to the command shown above results in an empty .txt file being created, and the three lines still printing to standard output.
I have tried the following as additions to the end of my command:
> log.txt
&> log.txt
>> log.txt
&>> log.txt
>> log.txt 2>&1
2>&1 | tee -a log.txt
As mentioned, all of these result in an empty log.txt and the three lines still printing to standard output.
I have read through tens of posts on StackOverflow, none of which seem to address this error. I know for certainty that my permissions are sufficient, and I am executing these in the bash shell.
I assume there is some aspect of bash that I am missing here, but I cannot figure it out. Thanks in advance.
As David C. Rankin pointed out, the output was being directed to stderr rather than stdout. Although several my initial redirections accounted for stderr, surrounding the entire first command with parentheses worked: