I created a pipeline that strings multiple programs together, unfortunately these programs are creating a huge amount of temporary files in the /tmp folder and when using large datasets my pipeline crashes because the /tmp folder fills up.
How do I export temporary files so that they are created in my current working directory where the pipeline is being run and not in the /tmp folder?
Currently I have tried to export the TMPDIR env variable to an already created directory /work in my current working directory, but the temporary files are still being created in the /tmp folder:
export TMPDIR=$(mktemp -d --tmpdir=/work)
<script>
rm -rf $TMPDIR
The programs do not have the option to set different output folders for temporary files created.
Just change
/worktoworkif the directoryworkis in your current directory./workmeans that you have a top-level directory named/work. Without the forward slash, it will be a relative directory.I just tested this code on my computer. No files were written to
/tmpthat I noticed: