snakemake: How to log python scripts executed by script directive?

2.1k Views Asked by At

Is there any way to easily save logs coming from snakemake rule executing python script with the script directive? The script uses libraries, which already have some integrated logging, and I want to store their logs. I do not want to use shell or run directive as they both are not so comfortable, when working with python scripts. In addition, I hope that the answer will ask for minimum changes in the python scripts, the main changes being in the Snakemake file. Here is the example of code:

rule correcting_ids:
    input:
        "assembly_{type}/{sample}/{dataset}/final.fasta"
    output:
        "assembly_{type}/{sample}/{dataset}/final.corrected.fasta"
    log:
        "logs/{sample}/correct_{type}_ids_{dataset}.log"
    script:
        "scripts/correcting_ids.py"```
2

There are 2 best solutions below

0
On

You could redirect all stdout and stderr to the logfile

python:

import sys

with open(snakemake.log[0], "w") as f:
    sys.stderr = sys.stdout = f
    [rest of the script]

R:

log <- file(snakemake@log[[1]], open="wt")
sink(log)
[rest of script]
1
On

I tried a solution where you can use the logggin module but still capture exceptions:


import sys,os
import logging, traceback
logging.basicConfig(filename=snakemake.log[0],
                    level=logging.INFO,
                    format='%(asctime)s %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    )
def handle_exception(exc_type, exc_value, exc_traceback):
    if issubclass(exc_type, KeyboardInterrupt):
        sys.__excepthook__(exc_type, exc_value, exc_traceback)
        return

    logger.error(''.join(["Uncaught exception: ",
                         *traceback.format_exception(exc_type, exc_value, exc_traceback)
                         ])
                 )
# Install exception handler
sys.excepthook = handle_exception