After running and stopping a parallel process, workers still appear to be running on my system and taking up resources. I know processes belong to parallel because they are the same number as the cores I had set up originally, and because upon inspecting in the system monitor, there are references to parallel. It seems like every time I go through my foreach loop, these processes accumulate and eventually slow down my system. Is there any programmatic way to get rid of them?
My code is basic, and I've tried doParallel, doFuture, and doSNOW.
Using Linux Mint Cinnamon OS here for development, and running in an HPC node using Linux where performance is much less than expected and I am showing that memory usage is much higher than CPU usage. My hunch is that these processes on my dev machine are the same thing that is slowing down the HPC.There should be no memory remaining after each foreach loop, so in my mind memory usage should be every efficient.
library(doFuture)
library(foreach)
my.cl <- parallelly::makeClusterPSOCK(n_cores, autoStop = TRUE)
doParallel::registerDoParallel(cl = my.cl, cores = n_cores)
foreach(...) %dofuture% {...}
parallel::stopCluster(cl = my.cl)
