mclapply using all cores but not all threads

Question

I have a 4 core Mac with 8 threads. My understanding of mclapply() is that it should utilize the 8 threads as processors, but when I run my script I only see 4 threads working. In my example, I am using a nested list because in my real work I have a 6 tier nested list that will take several days to run. I will be running my script on a 24 core Linux, so I am trying to understand how I can best utilize my resources.

library(zoo)

x = replicate(150000, rnorm(24)) 
list1 <- list(elem1 <- x, elem2 <- x)
nested_list <- list(elem1 = list1, elem2 = list1)

mclapply(nested_list, FUN = function(x){
  lapply(x, FUN = function(y){
    rollmean(y, 7)
  })
})

When I monitor my CPUs while running this script, I see this;

The even numbered cores do not seem to be working at all, and Core 1 seems to be to capable of doing more.

Is mclapply() not as simple as a drop-in replacement for lapply()? The actual functions that I am using come from specific packages that I need to use, so I cannot change functions to be more efficient (ie using dplyr functions instead of base functions). Please advise how I can use resources more efficiently.

Answer 1

Can mclapply and forking in general be limited by memory? I have a script that I am running on a 24 core Linux. At the beginning of the script, mclapply(, mc.cores = 18) was utilizing 18 cores. Half way through the script, after the workspace became very large (~3.5 GBs) only 2 cores would work when using mclapply(, mc.cores = 18).

I cleared up memory and this improved the forking. I tried mclapply(, mc.cores = 18) but from monitoring the CPU usage my core, I saw that still only 8 or 9 cores were working. I am wondering if this is still due to the memory needed to fork. My understanding is that when forking, a new session of R is created for each fork. Each of those sessions then needs to load the workspace/library, and this can be very memory intensive for large numbers of forks. Are there any ways to avoid this memory limitation when working with large data so as to better parallelize?

Here is the code I used to clear up memory;

# Clear plots
if(!is.null(dev.list())) dev.off()
# Clear console
cat("\014") 
# Clean workspace
rm(list=ls())

per http://www.sthda.com/english/articles/17-tips-tricks/75-clear-user-interface-and-free-memory-in-rrstudio/