Running script on my local computer when jobs submitted by qsub on a server finish

Question

I am submitting jobs via qsub to a server, and then want to analyze the results on the local machine after jobs are finished. Though I can find a way to submit the analysis job on the server, but don't know how to run that script on my local machine.

jobID=$(qsub job.sh)

qsub -W depend=afterok:$jobID analyze.sh

But instead of the above, I want something like

if(qsub -W depend=afterok:$jobID) finished successfully
sh analyze.sh
else
some script

How can I accomplish the above task?

Thank you very much.

Answer 1

I've faced a similar issue and I'll try to sketch the solution that worked for me:

After submitting your actual job,

jobID=$(qsub job.sh)

I would create a loop in your script that checks if the job is still running using

qstat $jobID | grep $jobID | awk '{print $5}'

Although I'm not 100% sure if the status is in the 5h column, you better double check. While the job is idling, the status will be I or Q, while running R, and afterwards C.

Once it's finished, I usually grep the output files for signs that the run was a success or not, and then run the appropriate post-processing script.

Answer 2

One thing that works for me is to use qsub synchronous with the option

qsub -sync y job.sh

(either on command line or as

#$ -sync y

in the script (job.sh) itself.

qsub will then exit with code 0 only if the job (or all array jobs) have finished successfully.