Tuning rocksDB to handle a lot of missing keys

Question

I'm trying to configure the rocksdb I'm using as a backend for my flink job. The state rocksdb needs to hold is not too big (around 5G) but it needs to deal with a lot of missing keys. I mean that 80% of the get requests will not find the key in the data base. I wonder whether there is a specific configuration to help with the memory consumption. I have tried to use bloom filters with 3 bits key and increase the block size to 16kb but it doesn't seem to help and the job fails on out of memory exceptions. I'll be glad to hear more suggestions

Answer 1

I wonder whether there is a specific configuration to help with the memory consumption.

If you are able to obtain a heap profiling (like https://gperftools.github.io/gperftools/heapprofile.html ?), it will be helpful to figure out out what part of RocksDB consume the most memory.

Given your memory budget (i.e, expectation) you plan for your RocksDB, you might start with some general memory controls as following:

• WriteBufferManager (for memtable, which is a large memory consumer) https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager
• Twisting block cache size (another large memory consumer) https://github.com/facebook/rocksdb/wiki/Block-Cache
• Tracking/capping memory in block cache (https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#block-cache, https://github.com/facebook/rocksdb/wiki/Projects-Being-Developed#improving-memory-efficiency)

I am not clear on how missing keys can potentially affect your memory consumption in specific way though.