How to preserve newline characters in CSH when storing in a variable or echo

Question

I have a python script which is getting called from a csh script ( I can not use bash). The python script prints out string with the newline characters and I am storing the output in a csh variable. However, as soon I store it in a variable, I am losing all the newline characters. Below are the scripts:

python script (pcmd.py)

def main(op):
    if op == "h": # print help
        # below newline character is getting lost
        print ("Help first line\n second line.")
    else:  # print and later execute csh commands
        print ("setenv SOME_ENV_VAR 123; setenv ANOTHER asd")

if __name__ == "__main__":
    import sys
    main(sys.argv[1])

csh script (pcmd.csh)

# store the py script in a variable
set pcmd="./pcmd.py $*"
# store the output of the first eval command
set output=`eval "$pcmd"`
# if output contains setenv, eval the output (csh commands)
if ( "$output" =~ *setenv* ) then
    eval "${output}"
else # otherwise, print the output
    echo "${output}"  # Issue : losing newline chars, tried echo -e too but of no avail

execution

pcmd.csh h
expects: Help first line
second line.

actual: Help first line second line.  # missing newline character

Can someone point out the missing bit in the scripts?

Answer 1

That's how csh behaves. From its man page:

Command substitution is indicated by a command enclosed in ``'. The output from such a command is broken into separate words at blanks, tabs and newlines, and null words are discarded. The output is variable and command substituted and put in place of the original string.

Command substitutions inside double quotes (`"') retain blanks and tabs; only newlines force new words. The single final newline does not force a new word in any case. It is thus possible for a command substitution to yield only part of a word, even if the command outputs a complete line.

By default, the shell since version 6.12 replaces all newline and carriage return characters in the command by spaces. If this is switched off by unsetting csubstnonl, newlines separate commands as usual.

The last time I used csh was the mid 1980's. I don't believe what you're trying to do is possible. Csh is a horrible shell. Even worse than POSIX shells like Bash. Even if you can't use Bash is there some reason you can't use /bin/sh? Or, better yet, a modern sane shell like Fish or Elvish?

One (partial) solution is to use double-quotes and array capture:

set output=("`$pcmd`")

That will make output an array var where each element is a line. You can then iterate over the array to construct a string where each element (line) is separated by a newline. Note that I omitted the eval because in your example it isn't necessary and is exceedingly dangerous. Note that I called this a "partial" solution because contiguous newlines are treated as if a single newline was present.