Executing external commands in parallel and capturing the output in an array in Rust

Question

I have the following while loop that runs generate_user_key for each of the file in the file_array, and outputs the result. I would like to parallelize this such that an array of the generated keys is returned, and the process is executed in parallel instead of sequential to make it faster.

use std::process::Command;

//file_array definition here

let mut i = 0;
while (i<100) {
  let generated_key = Command::new("generate_user_key")
                       .arg(file_array[i])
                       .output()
                       .expect("generate_user_key command failed to start");
  println!("stdout: {}", String::from_utf8_lossy(&generated_key.stdout));
  i=i+1;
}

What is the best way to implement this in rust?

Answer 1

If you want to loop over the array items using rayon then you can simply create into_par_iter and work on array items

use std::process::Command;
use rayon::iter::{ParallelIterator, IntoParallelIterator};

fn main() {
    let arr = [1, 2, 3, 4, 5];
    let result: Vec<_> = arr.into_par_iter().flat_map(|value| {
        let output = Command::new("sh")
                .args(["-c", &format!("echo {}", value)])
                .output()
                .expect("failed to execute process");
        println!("Index: {}, Output: {:?}", value, output.stdout);
        output.stdout
    });

    println!("{:?}", result);
}

You can also use range to loop over and use the counter as array index

use std::process::Command;
use rayon::iter::{ParallelIterator, IntoParallelIterator};

fn main() {
    let arr = [1, 2, 3, 4, 5];
    let result: Vec<_> = (0..arr.len()).into_par_iter().flat_map(|idx| {
        let output = Command::new("sh")
                .args(["-c", &format!("echo {}", arr[idx])])
                .output()
                .expect("failed to execute process");
        println!("Index: {}, Output: {:?}", idx, output.stdout);
        output.stdout
    });

    println!("{:?}", result);
}

Example using thread

use std::thread;
use std::time::Duration;

fn main() {
    let mut threads = vec![];
    for idx in 0..arr.len() {
        threads.push(thread::spawn(move || -> Vec<_> {
            let output = Command::new("sh")
                    .args(["-c", &format!("echo -n {}", idx)])
                    .output()
                    .expect("failed to execute process");
            println!("Index: {}, Output: {:?}", idx, output.stdout);
            thread::sleep(Duration::from_millis(1));
            output.stdout
        }));
    }

    let result = threads.into_iter().flat_map(|c| c.join().unwrap()).collect::<Vec<_>>();

    println!("{:?}", result);
}

Answer 2

When all else fails, throw threads at the problem. It almost certainly isn't the correct approach, but it works.

let mut join_handles = Vec::new();

for _ in 0..100 {
    join_handles.push(thread::spawn(|| {
        let generated_key = Command::new("generate_user_key")
                              .arg(file_array[i])
                              .output()
                              .expect("generate_user_key command failed to start");

        String::from_utf8_lossy(&generated_key.stdout)
    }));
}

let outputs = join_handles.into_iter().map(Result::unwrap).collect::<Vec<_>>();

Edit: The correct solution is probably using Command::spawn to start the processes without blocking. The OS can then handle running them in parallel and you can then collect the outputs.

Answer 3

This should be easy to do with rayon. E.g. something like this (untested since I don't have your generate_user_key):

use rayon::prelude::*;
let keys = (0..100).into_par_iter().map (|_| {
        Command::new("generate_user_key")
            .arg(file_array[i])
            .output()
            .expect("generate_user_key command failed to start")
            .stdout
    })
    .collect::<Vec<_>>();

or better:

use rayon::prelude::*;
let keys = file_array.par_iter().map (|f| {
        Command::new("generate_user_key")
            .arg(f)
            .output()
            .expect("generate_user_key command failed to start")
            .stdout
    })
    .collect::<Vec<_>>();