Kyotocabinet and Scala(Java): Limit of DB files open?

Question

I write scala application that use a lot of Kyotocabinet Db files (i need to open 500-3k little kyotocabinet files at one time).
But after 512 opened (created) db files i have a error "Error:success: no error", and new db file does not created.
After googling i found similar problem with Tokyo Cabinet + Java here: https://groups.google.com/forum/#!msg/tokyocabinet-users/ve6OsRm_hyU/hXC7795iqPsJ but without solution.
So what's the deal ? How can i open more kyotocabinet files in one application ? May be there is a some bug in Kyotocabinet ?

Answer 1

Ok it seems that i found the answer for my question...
This is not a Java or Scala specific issue but a Kyotocabinet
First i try to reproduce this behaviour in another language. So i writed test program using Perl , and it fails too, but with more informative message:
terminate called after throwing an instance of 'std::runtime_error'
what(): pthread_key_create

After that got sources of kyotocabinet and researched that for every kyoto File() object special TSDKey object created, and this object create pthread_key. By default one process can create limited number of this keys, and this number defined in PTHREAD_KEYS_MAX.
So it seems that open more than 512 kyoto db files in one process is not possible :(

Answer 2

Yes, I did a try. Increasing the value of macro PTHREAD_KEYS_MAX will solve the problem.

I'm using Ubuntu 16.04, with libc version 2.23. libpthread is a add on of libc.

1. Download source code from http://ftp.gnu.org/gnu/glibc/glibc-2.23.tar.bz2 .

2. Modify macro PTHREAD_KEYS_MAX from 1024 to 2048. Don't be too large. I have tried 10240, but the test program crashed.

3. Modify the maxcro value in system include file local_lim.h, and replace system libraries with your new version.

4. Rebuild kyotocabinet.

5. Copy libkyotocabinet.so and libpthread.so.0 to your test program path.

6. export LD_LIBRARY_PATH=./

7. Execute your test program. The number of max db files doubled.