Shared library is not recognized when using parallel foreach in R

61 Views Asked by At

I am using a shared library written in C in my R code. I load the compiled shared library using dyn.load command. I am going to call a shared library function in a parallelized foreach loop. Here is my code:

library(foreach)
library(doParallel)

totalCores = detectCores()
cluster <- makeCluster(totalCores[1]-1)
registerDoParallel(cluster)

dyn.load("package.so")

run <- function(i) {
    row <- data[i,]
    res <- .Call("c_function", as.double(row))
    return(res)
}


result <- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  run(i)
}

I get the following error:

Error in { :
  task 1 failed - "C symbol name "c_function" not in load table"

Although I have loaded the shared library, it seems c_function is not recognized in the parallel tasks. Of course when I let the dyn.load command in the foreach loop the problem is solved:

result <- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  dyn.load("package.so")
  run(i)
}

But I am not sure if this is the best practice since at each iteration the shared library (package.so) is loaded and it may be not efficient. Any ideas?

Edit:

Regarding to r2even's answer I tested the following code:

foreach(i=1:50,.packages='rootSolve') %dopar% {
  print(is.loaded("c_function"))
}

My PC has 10 CPU cores (20 threads) so the value of totalCores variable was 20 when I executed this code. Here is the result:

[[1]]
[1] TRUE

[[2]]
[1] TRUE

[[3]]
[1] TRUE

[[4]]
[1] TRUE

[[5]]
[1] TRUE

[[6]]
[1] TRUE

[[7]]
[1] TRUE

[[8]]
[1] TRUE

[[9]]
[1] TRUE

[[10]]
[1] TRUE

[[11]]
[1] FALSE

[[12]]
[1] FALSE

[[13]]
[1] FALSE

[[14]]
[1] FALSE

[[15]]
[1] FALSE

[[16]]
[1] FALSE

[[17]]
[1] FALSE

[[18]]
[1] FALSE

[[19]]
[1] FALSE

[[20]]
[1] TRUE

[[21]]
[1] TRUE

[[22]]
[1] TRUE

[[23]]
[1] TRUE

[[24]]
[1] TRUE

[[25]]
[1] TRUE

[[26]]
[1] TRUE

[[27]]
[1] TRUE

[[28]]
[1] TRUE

[[29]]
[1] TRUE

[[30]]
[1] TRUE

[[31]]
[1] TRUE

[[32]]
[1] TRUE

[[33]]
[1] TRUE

[[34]]
[1] TRUE

[[35]]
[1] TRUE

[[36]]
[1] TRUE

[[37]]
[1] TRUE

[[38]]
[1] TRUE

[[39]]
[1] TRUE

[[40]]
[1] TRUE

[[41]]
[1] TRUE

[[42]]
[1] TRUE

[[43]]
[1] TRUE

[[44]]
[1] TRUE

[[45]]
[1] TRUE

[[46]]
[1] TRUE

[[47]]
[1] TRUE

[[48]]
[1] TRUE

[[49]]
[1] TRUE

[[50]]
[1] TRUE

It raises several questions. Is the value of is.loaded("c_function") always False only in the the iterations 11 to 20? Is it guaranteed that from the iteration 21 to the rest it is always true?

1

There are 1 best solutions below

8
r2evans On

Try this *hack*, almost certainly less inefficient than repeated calls to dyn.load:

result <- foreach(i=1:nrow(data), .combine = rbind) %dopar% {
  if (!is.loaded("c_function")) dyn.load("package.so")
  run(i)
}

(Untested.)