I need to build a parallelized loop in R which is called from an external 3rd party S/W which fails upon silent errors. I.e. what does not show up in an interactive R session and can only be made visible by calling geterrmessage() is an issue for me which I am seeking to avoid.
A parallelized loop via future_apply sometimes causes a silent error all connections are in use (running the same code it sometimes is thrown and sometimes not). I've tried an alternative parallelization via foreach and it always produces a (different) silent error (see here). Despite great help from the community I've also not found a general approach how to avoid any silent errors so I am wondering if there is maybe a future.apply specific approach to avoid that.
The only thing I could find about it is this but I cannot derive a solution from there.
Below is a bit of code which has both the foreach and the future.apply example. Any idea why future_lapply sometimes throws that error or how to avoid it generally would be highly appreciated!
Thanks, Mark
# Create a matrix
num_rows <- 100
num_cols <- 500
mat <- matrix(nrow = num_rows, ncol = num_cols)
for (i in 1:num_cols) {
mat[, i] <- rnorm(num_rows)
}
# Load the required packages
library(foreach)
library(parallel)
library(doParallel)
library(future.apply)
print(geterrmessage())
# ----- nested loop via future -----
plan(multisession)
tnew <- future_lapply(1:ncol(mat), function(j) {
for (i in j:ncol(mat)) {
print(paste("j",j,"i",i))
}
})
print(geterrmessage())
plan(sequential)
# ----- nested loop via foreach -----
ncl <- max(2,floor(detectCores()*0.75)) #number of cores
clst <- makePSOCKcluster(n=ncl) #create cluster for R
registerDoParallel(cl = clst) #register cluster
t <- foreach (j=1:ncol(mat), .combine="c") %dopar% {
for (i in j:ncol(mat)) {
print(paste("j",j,"i",i))
}
}
print(geterrmessage())
stopCluster(cl=clst)