Last week I asked a question that caused by Kaa cluster. I deployed 3 Kaa clusters follow the kaa Cluster setup guide document. The databases are Cassandra and MariaDB . When I run C Client(I make a code change that every client can login in three kaa servers in random) . a lot of questions are reported in the Server log file. I try to solve these by myself.The forth error is:
2016-12-09 01:15:46,710 [pool-22-thread-1] ERROR o.k.k.s.thrift.NeighborConnection - EventWorker [192.168.10.14:9090:OPERATIONS_SERVICE:d9c71186-e45e-4c33-88b0-0ad5f26c6d17] error sending event messages pack.
com.twitter.common.thrift.TResourceExhaustedException: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use
at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:91) ~[thrift-0.0.64.jar:na]
at com.twitter.common.thrift.Thrift$1.invoke(Thrift.java:363) ~[thrift-0.0.64.jar:na]
at com.sun.proxy.$Proxy141.sendMessages(Unknown Source) ~[na:na]
at org.kaaproject.kaa.server.operations.service.event.DefaultEventService$MessageTemplate.process(DefaultEventService.java:727) ~[kaa-node-0.10.0.jar:na]
at org.kaaproject.kaa.server.thrift.NeighborConnection$EventWorker.run(NeighborConnection.java:322) ~[server-shared-0.10.0.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use
at com.twitter.common.thrift.ThriftFactory$ManagedThreadPool$1.rejectedExecution(ThriftFactory.java:237) ~[thrift-0.0.64.jar:na]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [na:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [na:1.8.0_111]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[na:1.8.0_111]
at com.twitter.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:66) ~[util-0.0.88.jar:na]
at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:71) ~[thrift-0.0.64.jar:na]
... 9 common frames omitted
I found the error is caused by Kaa parameter configuration.In kaa-node.properties file:
#Specify the max number of neighbor connections
max_number_neighbor_connections=3
If in kaa cluster , neighbor connections will become more and more .I made a test that modify this parameter value to 8 and run 12000 C Client .It also appeared lots of the same errors.Then I modify this parameter value to 50,it appeared one same error.So I modify this parameter value to 100,it didn't appear the same error.I hope Kaa team can verify this modification method and give a reasonable parameter value in cluster.