10 million puts failing in LevelDB

137 Views Asked by At

I just got started with leveldb and I was wondering if someone could answer a few questions about it for me. I am running it on a CentOs VM with 8 GB RAM, 20 GB storage, 2 cores, and an intel i7 processor. I don't know if that matters.

I wanted to test performance of leveldb so I attempted to do 10 million puts sequentially, (not in a batch) and I was not able to, I got an out of memory error. But when I did a top, I saw that node was only using about 25% of available memory. What am I doing wrong?

Here is my code:

var level = require('level');
var db = level('test.db' {
  valueEncoding: 'json'
});

for (i = 0; i <= 10000000; i++) {
  var value = { 'value': i };
  db.put(i.toString(), value);
}

When this failed, I tried 1 million puts which happened in 2-3 seconds without any problems. I upped it to 3.5 million and got an out of memory error after about 40 mins of waiting. I noticed that the code manages to go through all of the puts but then makes a bunch of *.ldb files in test.db and continues to make them until I run out of memory.

Could someone maybe explain to me whats going on?

0

There are 0 best solutions below