Waiting for a sub process in python which creates other processes

Question

I'm using python to launch an EnergyPlus batch simulation using "RunDirMulti.bat". This bat file creates one process for each simulation file on the directory. RunDirMulti looks like this:

@echo off
: This batch file is used to run EnergyPlus simulations using the RunEPlus.bat for all 
: the files in the current directory across multiple separate processor cores.  It has  
: two parameters, the weather file name to use for simulations and the number of 
: processors.
:
:   RunDirMulti <weather file> (opt) <number processor cores> (opt)
: 
: The RunDirMulti batch file loops through the files located in the current directory 
: and puts RunEPlus calls to each file into as many temporary batch files as processor  
: cores and then starts each of the batch files. No load balancing between the cores 
: is achieved using this method. The RunDirMulti.bat file should be located in a 
: directory that contains the IDF files.

: Main routine
: maindir - change if you did not install in default folder (to be safe, use quotes)
SET maindir="E:\PROGRAMAS\EnergyPlusV8-4-0\"
: The default weather file name if not provided as an argument.
SET weather=ECU_Quito.840710_IWEC
: The default number of separate processor cores that the simulations should use if 
: not provided as an argument.
SET numProc=4
IF "%1" NEQ "" SET weather=%1
IF "%2" NEQ "" SET numProc=%2
SET count=0
: Loop through the temporary directories and delete the temporary batch files.
for /L %%G in (1,1,%numProc%) do call :clean1 %%G
: Loop through each filename and divides them into each temporary batch file.
for %%F in (.\*.idf) do call :divide1  "%%F" 
: Loop through each temporary directory and adds EXIT to each temporary batch file.
for /L %%G in (1,1,%numProc%) do echo EXIT >> .\tempsim%%G\simpart%%G.bat
: Loop through each temporary directory and starts the batch file in a new window 
for /L %%G in (1,1,%numProc%) do call :startEach1 %%G
: The following line goes to the end of the batch file.
GOTO:eof


: Subroutine that deletes the temporary batch files from each
: working directory.
:clean1
IF EXIST .\tempsim%1\simpart%1.bat (
  DEL .\tempsim%1\simpart%1.bat
) ELSE (
  MD .\tempsim%1
)
: The following line returns to the main routine.
GOTO:eof


: Subroutine that takes the file name and uses a counter
: and the MOD operator (double percent) to group them
: into as many batch files as necessary.
:divide1
SET /a count="count + 1"
SET /a group="count %% numProc + 1"
echo CALL %maindir%RunEPlus.bat "%~dpn1"     %weather% >>.\tempsim%group%\simpart%group%.bat 
: The following line returns to the main routine.
GOTO:eof


: Subroutine that starts each batch file
:startEach1
cd .\tempsim%1
START "Batch Simulation%1" simpart%1.bat
cd ..
: The following line returns to the main routine.
GOTO:eof

In my script I'm using this

p=subprocess.check_call("RunDirMulti.bat")

but it does not wait for all the child process to finish. I've already tried Popen and Popen.wait(). Can some one give me an idea please?

Answer 1

Solved it creating a function that checks if cmd is open (in my case i'm running ipython notebook so there is already one cmd open all the time).

def running():
    n=0 #number of cmd instances running 
    prog=[line.split() for line in subprocess.check_output("tasklist").splitlines()]
    [prog.pop(e) for e in [0,1,2]] #useless 
    for task in prog:
        if task[0]=="cmd.exe":
            n=n+1
    if n>1:
        return True
    else:
        return False

This should work with any program.

Answer 2

When the process is complete you could output a done file via your batch script:

type NUL > done.txt

Then add something like the following to your python script

import os
import time

done_file_path = r'C:\Users\user\done.txt'
while not os.path.exists(done_file_path):
    print "Done file not created, Sleeping for 10 seconds"
    time.sleep(10) # Sleep 10 second
os.remove(done_file_path) # Remove Done file, so next run is successful

Adjust the path(C:\Users\user\done.txt) to where you would be outputting the done file of course.