How to free memory which is used by big.matrix objects of crashed R sessions

Question

I use the bigmemory package to access big matrix objects in parallel, e.g. like this

a <- bigmemory::big.matrix(nrow = 200, ncol = 100, shared = TRUE) # shared = TRUE is the default

However, working with the resulting objects sometimes cause R to crash. This means that the memory used by the matrix objects is not released. The bigmemory manual warns of such a case but presents no solution:

Abruptly closed R (using e.g. task manager) will not have a chance to finalize the big.matrix objects, which will result in a memory leak, as the big.matrices will remain in the memory (perhaps under obfuscated names) with no easy way to reconnect R to them

After a few crashes and restarts of my R process, I get the following error:

No space left on device 
Error in CreateSharedMatrix(as.double(nrow),
as.double(ncol), as.character(colnames),  : 
  The shared matrix could not be created

Obviously, my memory is blocked by orphaned big matrices. I tried the command ipcs, which is advertised to list shared memory blocks, but the sizes of the segments listed there are much too small compared to my matrix objects. This means also that ipcrm is of no use here to remove my orphaned objects.

Where does bigmemory store its objects on different operating systems and how do I delete orphaned ones?

Answer 1

Linux

A call to df -h solved the mystery for my operating system (Linux/CentOS).

$ df -h

Filesystem       Size  Used Avail Use% Mounted on
... 
tmpfs           1008G 1008G     0 100% /dev/shm
...

There is a temporary file system in the folder /dev/shm. Files therein exist only in RAM. This file system is used to share data between processes. In this folder were several files with random strings as names, and multiple files with the same prefix, which seem to be related to the same big.matrix object:

$ ls -l /dev/shm

-rw-r--r-- 1 user  grp 320000 Apr 26 13:42 gBDEDtvwNegvocUQpYNRMRWP
-rw-r--r-- 1 user  grp      8 Apr 26 13:42 gBDEDtvwNegvocUQpYNRMRWP_counter
-rw-r--r-- 1 user  grp     32 Apr 26 13:42 sem.gBDEDtvwNegvocUQpYNRMRWP_bigmemory_counter_mutex

Unfortunately, I don't know which matrix belongs to which file, but if you have no R processes running at the time, deleting files with this name pattern should remove the orphaned objects.

Windows

I don't know how other OS's do this, so feel free to add it into this community wiki if you know