There is a Java Springboot application that connects to PostgreSQL in our project and we have HikariCP connection pool. After a few hours, the active number of operations between the Java application and the PostgreSQL server increases
In these situations we get errors like this
2021-10-23 23:09:28.703-04:00 DEBUG [HikariPool-1 housekeeper] com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=4, idle=6, waiting=0)
2021-10-23 23:09:28.703-04:00 DEBUG [HikariPool-1 housekeeper] com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Fill pool skipped, pool is at sufficient level.
2021-10-23 23:09:45.937-04:00 DEBUG [CoreTaskScheduler1] com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Timeout failure stats (total=9, active=5, idle=4, waiting=0)
2021-10-23 23:09:45.953-04:00 DEBUG [HikariPool-1 connection closer] com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection org.postgresql.jdbc4.Jdbc4Connection@2ce66f: (connection is dead)
2021-10-23 23:09:46.016-04:00 ERROR [CoreEventBus-0] f.n.n.s.impl.core.CoreEventHandler - Database connection pool lost connection to DB.
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 5791ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) ~[HikariCP-4.0.3.jar:na]
at com.zaxxer.hikari.metrics.dropwizard.CodahaleHealthChecker$ConnectivityHealthCheck.check(CodahaleHealthChecker.java:95) ~[HikariCP-4.0.3.jar:na]
at com.codahale.metrics.health.HealthCheck.execute(HealthCheck.java:172) ~[metrics-healthchecks-3.1.4.jar:3.1.4]
at com.codahale.metrics.health.HealthCheckRegistry.runHealthCheck(HealthCheckRegistry.java:66) ~[metrics-healthchecks-3.1.4.jar:3.1.4]
at x.y.z.t.u.v.database.DatabaseConnectionHook.runHealthCheck(DatabaseConnectionHook.java:57)
at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_66]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_66]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_66]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_66]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_66]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
Here is the version info:
Spring version: 4.3.9.RELEASE
HikariCP version: 4.0.3
Hibernate version: 5.2.10.Final
Postgresql jdbc: 9.3-1104-jdbc41
Server version: Jetty/9.4.11
JVM Version: 1.8.0_19
Here is the Hikari pool configuration:
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - HikariPool-1 - configuration:
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - autoCommit......................true
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - catalog.........................null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - connectionInitSql...............null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - connectionTestQuery.............null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - dataSource......................null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - dataSourceClassName.............null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>}
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - driverClassName................."org.postgresql.Driver"
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{connectivityCheckTimeoutMs=1000}
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............com.codahale.metrics.health.HealthCheckRegistry@11c9c56
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - initializationFailFast..........false
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - jdbc4ConnectionTest.............false
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - jdbcUrl........................."jdbc:postgresql://127.0.0.1:5432/db_log"
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........0
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - maxLifetime.....................1800000
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - maximumPoolSize.................10
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - metricRegistry..................null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - minimumIdle.....................10
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - password........................<masked>
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-1"
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - readOnly........................false
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - registerMbeans..................false
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - scheduledExecutorService........null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - threadFactory...................null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - transactionIsolation............null
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - username........................"log_user"
2021-10-24 15:45:32.521+00:00 DEBUG [WrapperListener_start_runner] com.zaxxer.hikari.HikariConfig - validationTimeout...............5000