Silent errors in parallelized R loops with future.apply (...and foreach)

22 Views Asked by At

I need to build a parallelized loop in R which is called from an external 3rd party S/W which fails upon silent errors. I.e. what does not show up in an interactive R session and can only be made visible by calling geterrmessage() is an issue for me which I am seeking to avoid.

A parallelized loop via future_apply sometimes causes a silent error all connections are in use (running the same code it sometimes is thrown and sometimes not). I've tried an alternative parallelization via foreach and it always produces a (different) silent error (see here). Despite great help from the community I've also not found a general approach how to avoid any silent errors so I am wondering if there is maybe a future.apply specific approach to avoid that. The only thing I could find about it is this but I cannot derive a solution from there.

Below is a bit of code which has both the foreach and the future.apply example. Any idea why future_lapply sometimes throws that error or how to avoid it generally would be highly appreciated!

Thanks, Mark

# Create a matrix
num_rows <- 100
num_cols <- 500
mat <- matrix(nrow = num_rows, ncol = num_cols)
for (i in 1:num_cols) {
    mat[, i] <- rnorm(num_rows)
  }

# Load the required packages
library(foreach)
library(parallel)
library(doParallel)
library(future.apply)

print(geterrmessage())

# ----- nested loop via future -----
plan(multisession)
tnew <- future_lapply(1:ncol(mat), function(j) {
    for (i in j:ncol(mat)) {
        print(paste("j",j,"i",i))
    }
})
print(geterrmessage())
plan(sequential)


# ----- nested loop via foreach -----
ncl <- max(2,floor(detectCores()*0.75)) #number of cores
clst <- makePSOCKcluster(n=ncl) #create cluster for R
registerDoParallel(cl = clst) #register cluster
t <- foreach (j=1:ncol(mat), .combine="c") %dopar% {
        for (i in j:ncol(mat)) {
            print(paste("j",j,"i",i))
        }
    }
print(geterrmessage())
stopCluster(cl=clst)
0

There are 0 best solutions below