Problem with big_cprodMat output: Matrix too large

Question

I am trying to get the cross-product of two "Large matrices" in R. Unfortunately, the data cannot be disclosed here, but the dimensions are 177304*23, and 23*177304, so that the results should have the dimensions 177304*177304.

As my RAM is 8 GB, I have been using the FBM-class to store the matrices in the Disk Storage Space instead. But when I use the function big_cprodMat to find the crossproduct, it seems to first solve the product using a matrix class-object as the output and R crashes instead.

Is there anything I could do to avoid this?

I'm new to the website and greatly appreciate any help anyone could offer.

Answer 1

Using a smaller example:

N <- 17e3
M <- 23
A <- matrix(rnorm(N * M), N, M)
B <- matrix(rnorm(N * M), M, N)

You can fill the resulting matrix by blocks (note that it will take 234 GB on disk for the size you ask for):

library(bigstatsr)
AB <- FBM(N, N, backingfile = "test")$save()
big_apply(AB, a.FUN = function(X, ind) {
  X[, ind] <- A %*% B[, ind]
  NULL
}, block.size = 1e3)

Do not hesitate to open an issue on the GitHub repo of {bigstatsr}.