> 60K inserts per second key value store to be used from Java

Question

Am working on custom solution where I need to store huge data to file/db at the speed of 60K records per second. This data is result of incoming continuous stream.

1. leveldb - it can NOT be accessed simultaneously from multiple java processes

2. berkleydb/kyoto cabinet : prohibitive commercial license

3. sqlite : Tried sqlite4java but it doesnt support bulk operations and is not performing at the required speed. Also tried jdbc wrapper(http://www.zentus.com/sqlitejdbc/) this also doesnt perform at the required speed.

Can someone please suggest me solution which will just allow me to dump data to db/file and can be accessed by multiple processes(+threads)?

Answer 1

Can someone please suggest me solution which will just allow me to dump data to db/file and can be accessed by multiple processes(+threads)?

If you want access from multiple processes, multiple threads and/or both, then it's very difficult to reach your goal of 60k records per second simply because disk I/O is not very efficient when you're using multiple threads/processes. Any solution that you have would need to reduce to the same basic model: only one thread can write to LevelDB.

If you want cross-process access to LevelDB, then you can write a very simple communication layer between your processes using sockets or pipes. You would dedicate a single process to accept multiple socket or pipe connections from all the client processes and you would accept streaming data. You then write your data to LevelDB from that process- let's call that the data process.

The data process will have multiple threads that supply data, but only one reading/writing thread that reads/writes data into LevelDB, so it might also be a good idea to use a BlockingQueue which the client threads would enqueue records to and the data thread will read records from (blocking if there is no data).

Answer 2

You could try MySQL InnoDB. I can't say if its fast enough for you, but multiple processes can access different tuples of data (locking is per row).

Answer 3

Try Redis, is very fast, you have the possibility to save your data on ram and later to disk, good solution for you problem i think.

Here the clients lib for java.

Answer 4

Try sqlite-jdbc library http://code.google.com/p/sqlite-jdbc/ It is a JDBC wrapper on native dll/so implementations.

Answer 5

Try RocksDB by Facebook. It can be accessed from java and from multiple threads. It has better performance than LevelDB.