How to append stdout and stderr to file in specified directory using CWL (Common Workflow Language)?

Question

I have been try to write in CWL a description that appends stdout and stderr to a file in a specific directory, something like printf 'some message' >> out_directory/out_file 2>&1

With the following CWL description:

# File: test.cwl
cwlVersion: v1.0
class: CommandLineTool

requirements:
- class: InlineJavascriptRequirement

baseCommand: [printf]

inputs:
  message:
    type: string
    inputBinding:
      position: 1

  output_name: 
    type: string

  output_dir: 
    type: string  

stdout: $(inputs.output_name)
stderr: $(inputs.output_name)

outputs:
  output:
    type: File
    outputBinding:
      glob: $(inputs.output_dir)/$(inputs.output_name)

and input file:

# File: test.yml
message: 'some message'
output_name: out_file
output_dir: out_directory

I get this error

Error collecting output for parameter 'output':
test.cwl:28:7: Did not find output file with glob pattern: '['out_directory/out_file']'

Any ideas about printing to specific directories? Also, how can I use >> instead of > ?

Thank you!

Answer 1

Take a look at the section “Capturing Standard Output” of the User Guide: https://www.commonwl.org/user_guide/05-stdout/index.html. It has an example of how to use the stdout variable.

In your example, I think you don't need to specify the output type as File nor its binding when using stdout. Instead you can have this as your workflow file.

# File: test.cwl
cwlVersion: v1.0
class: CommandLineTool

requirements:
- class: InlineJavascriptRequirement

baseCommand: [printf]

inputs:
  message:
    type: string
    inputBinding:
      position: 1

  output_name: 
    type: string
 

stdout: $(inputs.output_name)
stderr: $(inputs.output_name)

outputs:
  output:
    type: stdout

And this as the input:

# File: input.yaml
message: 'some message'
output_name: out_file

You don't need to pass an input parameter for the output directory. By default it will end up in your local directory with your workflow, but you can change it with the following command:

(venv) kinow@ranma:/tmp$ cwltool --outdir=out_directory test.cwl input.yaml
INFO /home/kinow/Development/python/workspace/cwltool/venv/bin/cwltool 3.1.20211004060744
INFO Resolved 'test.cwl' to 'file:///tmp/test.cwl'
INFO [job test.cwl] /tmp/gpsyzk7h$ printf \
    'some message' > /tmp/gpsyzk7h/out_file 2> /tmp/gpsyzk7h/out_file
INFO [job test.cwl] completed success
{
    "output": {
        "location": "file:///tmp/out_directory/out_file",
        "basename": "out_file",
        "class": "File",
        "checksum": "sha1$af52b1a96761839824b7b4c0e6cea4b09f2b0710",
        "size": 12,
        "path": "/tmp/out_directory/out_file"
    }
}
INFO Final process status is success
(venv) kinow@ranma:/tmp$ cat out_directory/out_file 
some message

And that way your out_file should appear in ./out_directory.

For future questions about CWL, you may want to post it to the CWL Discourse forum instead: https://cwl.discourse.group/

-Bruno

Answer 2

I finally managed to move the output to a specific directory by following the answer here

outputs:
  output:
    type: stdout
  output_directory:
    type: Directory
    outputBinding:
      glob: .
      outputEval: |
        ${
          self[0].basename = inputs.output_dir;
          return self[0]
        }