using pipe to run a file and append result as a new col

Question

I have a CSV file with column A, B, C. I have a script that will read 2 sentences and output a score. Specifically, i need the script to read col B and col C to output a score.

For example:

A      B       C
id  string1 string2

My script works this way: myscript(string1, string2) = score

How do i write a line of code that takes only column B and C and then output everything into a new file?

I want to have the output into a new file like this:

A       B        C       D
id   string1   string2  score

I believe the code will go something like this but not sure of the right syntax.

textfile.csv | perl myscript.pl > new_textfile.csv

Please help

Answer 1

I believe this is what you need

 $ awk -F "," '{print $2,",",$3;}' textfile.csv | perl myscript.pl > new_textfile.csv

Answer 2

The standard Unix tool for "adding columns" is paste.

If your script generates one line of output for each line of input in textfile.csv, you could paste the files together side-by-side like this:

perl myscript.pl > tmpfile.csv
paste -d , textfile.csv tmpfile.csv > new_textfile.csv

Or you could do this in one step (omitting the temporary file) by using a pipe instead:

perl myscript.pl | paste -d , textfile.csv - > new_textfile.csv

If your script operates only on columns 2 and 3 of the input, and doesn't know how to ignore column 1, you could extract columns 2 and 3 to feed into your script:

awk -F, '{print $2, $3}' textfile.csv | perl myscript.pl | ...

(But beware that awk invoked with -F, is not a fully-general CSV parser, and won't handle quoting.)

Finally, if each invocation of your script only knows how to operate on one pair of numbers (that is, on one line of input from textfile.csv), you could use a loop, something like this:

cp /dev/null tmpfile.csv
awk -F, '{print $2, $3}' textfile.csv | while read col2 col3
do
    perl myscript.pl $col2 $col3 >> tmpfile.csv
done
paste -d , textfile.csv tmpfile.csv > new_textfile.csv

Or you could get fancy and redirect the output of the loop:

awk -F, '{print $2, $3}' textfile.csv | while read col2 col3
do
    perl myscript.pl $col2 $col3
done > tmpfile.csv
paste -d , textfile.csv tmpfile.csv > new_textfile.csv

Or if you really wanted to be a cowboy:

awk -F, '{print $2, $3}' textfile.csv | while read col2 col3
do
    perl myscript.pl $col2 $col3
done | paste -d , textfile.csv - > new_textfile.csv