First, everything ok. After amost 40h, I receive a lot of error
org.springframework.dao.DataAccessResourceFailureException: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.spi.R2dbcTimeoutException: Connection Acquisition timed out after 5000ms
at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88) ~[spring-r2dbc-5.3.6.jar!/:5.3.6]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ Handler cn.booslink.controller.DeviceController#login(String, String) [DispatcherHandler]
Stack trace:
at org.springframework.r2dbc.connection.ConnectionFactoryUtils.lambda$getConnection$0(ConnectionFactoryUtils.java:88) ~[spring-r2dbc-5.3.6.jar!/:5.3.6]
at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3488) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxRetry$RetrySubscriber.onError(FluxRetry.java:94) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.Operators.error(Operators.java:197) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.MonoError.subscribe(MonoError.java:52) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.Mono.subscribe(Mono.java:4150) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:294) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:279) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:418) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:119) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [reactor-core-3.4.5.jar!/:3.4.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: io.r2dbc.spi.R2dbcTimeoutException: Connection Acquisition timed out after 5000ms
at io.r2dbc.pool.ConnectionPool.lambda$null$6(ConnectionPool.java:123) ~[r2dbc-pool-0.8.6.RELEASE.jar!/:0.8.6.RELEASE]
at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3474) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.Mono.lambda$onErrorResume$32(Mono.java:3564) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:294) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:279) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:418) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:119) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [reactor-core-3.4.5.jar!/:3.4.5]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [reactor-core-3.4.5.jar!/:3.4.5]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 5000ms in 'Connection Acquisition from [MariadbConnectionFactory{configuration=MariadbConnectionConfiguration{datbase='xx', host='xx', connectTimeout=PT10S, socketTimeout=null, tcpKeepAlive=false, tcpAbortiveClose=false, password=***********, port=3306, prepareCacheSize=250, socket='null', username='x'xxx', allowMultiQueries=false, allowPipelining=true, connectionAttributes=null, sessionVariables=null, sslConfig=SslConfig{sslMode=DISABLED, serverSslCert='null', clientSslCert='null', tlsProtocol=null}, rsaPublicKey='null', cachingRsaPublicKey='null', allowPublicKeyRetrieval=false, isolationLevel=IsolationLevel{sql='REPEATABLE READ'}, useServerPrepStmts=false, autocommit=true, pamOtherPwd=}}]' (and no fallback has been configured)
... 13 common frames omitted
It seems like connection pool can't acquire connection now. Bug of r2dbc-pool? Or mysql limit??
Not easy to reproduce, normally the peak will be at 19:00
Need help