I have a very large matrix (16,000 by 16,000) that I need to upload into a MongoDB database. I've tried to split this matrix into a list and then use parLapply to convert to BSON. Here's my code
library(foreach)
library(doMC)
library(parallel)
noCores <- detectCores() - 1
registerDoMC(noCores)
matrix <- as.matrix(cov(mtcars))
cl <- makeCluster(mc <- getOption("cl.cores", detectCores()/2))
clusterEvalQ(cl, {
library(rmongodb)
library(rjson)})
lst <- split(as.data.frame(matrix), rownames(matrix))
df_list <- parLapply(cl, lst, mongo.bson.from.list)
I'm using mtcars as a test matrix here. I get this error
> df_list
$am
Error in print.mongo.bson(0L) :
mongo.bson object appears to have been destroyed
Seems like parLapply isn't creating the list correctly. Any ideas? Thank you in advance.