How to split a large matrix into a BSON list using parLapply and rmongodb?

68 Views Asked by At

I have a very large matrix (16,000 by 16,000) that I need to upload into a MongoDB database. I've tried to split this matrix into a list and then use parLapply to convert to BSON. Here's my code

library(foreach)
library(doMC)
library(parallel)

noCores <- detectCores() - 1
registerDoMC(noCores)

matrix <- as.matrix(cov(mtcars))

cl <- makeCluster(mc <- getOption("cl.cores", detectCores()/2))
clusterEvalQ(cl, {
library(rmongodb)
library(rjson)})

lst <- split(as.data.frame(matrix), rownames(matrix))
df_list <- parLapply(cl, lst, mongo.bson.from.list) 

I'm using mtcars as a test matrix here. I get this error

> df_list
$am
Error in print.mongo.bson(0L) : 
mongo.bson object appears to have been destroyed

Seems like parLapply isn't creating the list correctly. Any ideas? Thank you in advance.

0

There are 0 best solutions below