Berkeley DB JE JDB files keep increasing

Question

I am debugging a disk space issue in my system and found that .jdb files are consuming most of the space.

while browsing, found this link https://backstage.forgerock.com/knowledge/kb/article/a14630082, In my case lnSizeCorrectionFactor was around 1.4 and fileDeleted=false. I ran the disk space command to find the space utilization, it turned out to be non-zero for all the jdb files but most of the files have utilization as single digit values ranging from 2-9.

I am using je-5.0.58 version, which is not latest. My question is according to BDB doc https://docs.oracle.com/cd/E17277_02/html/GettingStartedGuide/backgroundthreads.html#cleaner, jdb file needs to be cleaned if it reaches lesser than default value of 50%. In my case even they are at single digit, why they are not getting cleaned?

I didn't explicitly set any environment config so it should be using default values by definition. Below is the code to create a bdb repository.

 private static Repository createBDBRepository(File environmentHome) throws 
 RepositoryException {
            BDBRepositoryBuilder builder = new BDBRepositoryBuilder();
            builder.setName("localbdb");
            builder.setEnvironmentHomeFile(environmentHome);
            builder.setTransactionNoSync(false);

            // Set BDB-JE flavor
            builder.setProduct("JE");

            builder.setCacheSize(20 * 1024 * 1024L);
            return builder.build();
    }

Log line from je.info file

   Chose lowest utilized file for cleaning. fileChosen: 0x50cbecc totalUtilization: 49 bestFileUtilization: 8 lnSizeCorrectionFactor: 1.1012049 isProbe: false
   No file selected for cleaning. totalUtilization: 50 bestFileUtilization: 8 lnSizeCorrectionFactor: 1.1012049 isProbe: false

Answer 1

As the ForgeRock document describes, this is due to a bug in the version 5 of BDB JE, which has been resolved with version 6 and higher.

Quoting BDB JE 6.x release notes:

An improvement was made to the calculation of log utilization to avoid under-cleaning or over-cleaning. For example, when log utilization was estimated to be lower than actual utilization, unnecessary over-cleaning would occur, which could reduce performance. Or when log utilization was estimated to be higher than actual utilization, under-cleaning would prevent reclaiming unused disk space. To prevent these problems, the size of each logged record is now stored in the Btree BINs (bottom internal nodes), so that utilization can be calculated correctly during record updates and deletions, while still avoiding a fetch of the old version of the record. With this change, the utilization adjustment facility in the log cleaner, which attempted to compensate for this problem by estimating utilization, is no longer needed by most applications.

Therefore the EnvironmentConfig.CLEANER_ADJUST_UTILIZATION parameter is now false by default rather than true, and will be disabled completely in a future version of JE. For more information, see the javadoc for this parameter.

[#22275] (6.0.7)