From CMD, is it possible to execute a command after a for loop using start to launch several parallel tasks, but only AFTER all tasks have completed?

Question

I'm running a command that modifies all the audio files in a folder, due to a bug in ffmpeg, I need to chain together an ffmpeg and sox command, but sox is really slow, so I run them in parallel, which dramatically speeds up the process (makes it about 10 times faster, probably the more cores in your CPU, the faster it will go). That all works fine.

The problem is that because those are all spawned as separate tasks, it returns and runs the commands after the FOR loop before it has completed the tasks. In my case, that's to delete the intermediate temp files (del int_*.wav in the example below). Here's a simplified example (removed most parameters to make it easier to read):

md "Ready" & (for %x in ("*.mp3") do (start "Convert" cmd /c "ffmpeg -i "%x" -f wav "int_%x.wav" & sox "int_%x.wav" -r 44100 "Ready\ready-%x"")) && del int_*.wav

That converts all the MP3 files in the current directory per my parameters to a different set of MP3 files in the destination directory (the Ready subdirectory). It uses .WAV files as intermediaries because those are lossless and fast.

The above runs correctly, except for the last part, del int_*.wav, because it tries to run that before all the START threads have finished, and so it does nothing.

It errors with:

Could Not Find C:\<parent directory name>\int_*.wav

Which is expected, because it gets to the DEL before it has created the files to delete. Hence my need to have it pause until the FOR loop has completed and all the START tasks have closed.

It does run the DEL command correctly if I leave out the START command and don't try to run them in parallel, but then it's very slow (this is how I originally did it, then came up with the above solution to make it faster):

md "Ready" & (for %x in ("*.mp3") do (ffmpeg -i "%x" -f wav "int_%x.wav" & start sox "int_%x.wav" -r 44100 "Ready\ready-%x")) & del int_*.wav

And note that this example only has the DEL command at the end, but in my actual use case, I have multiple commands after the loop, including a batch file, all chained together with & signs. So I can't use a solution that just provides a different way to delete the files. I need to have everything after the double closing parentheses wait to run until the loop has completed and all the started tasks have closed.

Also note that I'm intentionally running this as one command from the command prompt rather than via a batch file. This is because I have to change a small varying subset of many parameters and generally run it on a subset of the files in a directory. So in this particular case, it's much easier to just paste the command and tweak the relevant parameters and filenames with wildcards than make a batch file and have to pass it everything every time. I'd like to keep it as a single CMD line if possible.

Is there any way to do that?

Answer 1

Avoid cmd these days. Everything is much easier in PowerShell, and many things that are impossible to do in cmd can be achievable in PowerShell

foreach (f in Get-ChildItem -Filter "*.mp3") {
    Start-Job -ScriptBlock {
        ffmpeg -i "$f" -f wav "int_$f.wav"
        sox "int_$f.wav" -r 44100 "Ready\ready-$f"
    }
}
Get-Job | Wait-Job
Remove-Item int_*.wav

Or you can pipe directly without Get-Job like this

foreach (f in ls "*.mp3") { sajb {
    ffmpeg -i "$f" -f wav "int_$f.wav"; sox "int_$f.wav" -r 44100 "Ready\ready-$f"
} } | wjb
del int_*.wav

Or even make it a one-liner

foreach (f in ls "*.mp3") { sajb { ffmpeg -i "$f" -f wav "int_$f.wav"; sox "int_$f.wav" -r 44100 "Ready\ready-$f" } } | wjb; rm int_*.wav

---

So in this particular case, it's much easier to just paste the command and tweak the relevant parameters and filenames with wildcards than make a batch file and have to pass it everything every time

Nothing prevents you from setting a default value if the parameter is not set. This is possible in any shell scripting languages. Just pass the arguments that you need to change. However in batch you'll need to read arguments and set the flags manually while in PowerShell just define what you want with Param() and everything will be handled automatically

---

Alternatively you can use Start-Process/Wait-Process instead of Start-Job/Wait-Job but this is longer, less readable and may not work for some commands

Start-Process -PassThru {
    cmd /c "ffmpeg -i `"$f`" -f wav `"int_$f.wav`" & sox `"int_$f.wav`" -r 44100 `"Ready\ready-$f`""
} | Wait-Process
rm int_*.wav

Answer 2

[untested]

for %%e in ('dir int_* 2^>nul^|find /c "int") do set starting=%%e
set /a converted=0
md "Ready" & (for %x in ("*.mp3") do (set /a converted+=1&ffmpeg -i "%x" -f wav "int_%x.wav" & start sox "int_%x.wav" -r 44100 "Ready\ready-%x"))
:wait
timeout /t 1 >nul
for %%e in ('dir int_* 2^>nul^|find /c "int") do set /a current=%%e-starting
if %current% neq %converted% goto wait

This shoud set starting to the number of int_* files originally in the directory, then count the conversions to be done in converted.

Then calculate the number completed in current and if not the same as the number in converted, wait another 1 second.

Answer 3

PowerShell does seem to provide the best solution to this, because the wjb option dutifully waits for the running parallel tasks to all finish, then immediately runs the delete. Thanks to @phuclv for steering me down this path. However, this requires a few additional steps:

First, upgrade to PowerShell 7 (only 5.2 included by default with Windows 11, which has limits and problems with its parallel / StartJob functions. Either run from Microsoft Store (search for PowerShell) or, at a PS prompt, run: winget install --id Microsoft.Powershell --source winget

Then launch a new PowerShell 7 Terminal window. The old PowerShell version 5 will still be installed alongside version 7. To open the new one, launch a new Terminal window, then open a new tab, selecting "PowerShell" instead of "Windows PowerShell" (Windows Powershell is the old version 5).

Now you can run a command with the "ForEach-Object -parallel" like this:

md "Ready"; Get-ChildItem -Filter "*.mp3" | ForEach-Object -parallel { ffmpeg -i $.name -f flac -af dynaudnorm -id3v2_version 3 ($.name+".flac"); sox --norm=-3.25 ($.fullname + ".flac") -c 2 -C 192 -r 44100 (".\Ready\ready-" + $.name) } | wjb; rm *.flac

Additional considerations specific to PowerShell:

While you can use the StartJob function under version 5, that spawned tasks lose awareness of the current directory. In place of $, you can use ($using:), but I had trouble getting this to work (upgrading to version 7 and using -parallel was easier).

There is also a newer Start-ThreadJob which may be faster and more efficient than Start-Job, but at least for my purposes, the ForEach-Object -parallel" option worked perfectly.