Unexpected connection timeout in datastax cassandra -com.datastax.shaded.netty.channel.ConnectTimeoutException

Question

We are getting connection in Session object at very first time application starts . Our scheduler keeps inserting data into Cassandra using that . but some point of time , its getting connection timeout and then its reconnecting with Cassandra and application works normally .

Frequent timeout at scheduler . (I have changed my ip to ... in this post)

2015-08-19 09:19:36 DEBUG Connection:274 - Defuncting connection to /*.*.*.*:9042
com.datastax.driver.core.TransportException: [/10.0.*.*:9042] Cannot connect
    at com.datastax.driver.core.Connection.<init>(Connection.java:104)
    at com.datastax.driver.core.Connection$Factory.open(Connection.java:544)
    at com.datastax.driver.core.Cluster$Manager$5.tryReconnect(Cluster.java:1652)
    at com.datastax.driver.core.AbstractReconnectionHandler.run(AbstractReconnectionHandler.java:124)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    ... 3 more
2015-08-19 09:19:36 DEBUG Connection:449 - Connection[/*.*.*.*:9042-8, inFlight=0, closed=true] closing connection
2015-08-19 09:19:36 DEBUG Connection:721 - Connection[/*.*.*.*:9042-8, inFlight=0, closed=false] connection error
com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
2015-08-19 09:19:36 DEBUG Cluster:1681 - Failed reconnection to /10.0.1.18:9042 ([/*.*.*.*:9042] Cannot connect), scheduling retry in 128000 milliseconds
2015-08-19 09:19:36 DEBUG Connection:274 - Defuncting connection to /*.*.*.*:9042
com.datastax.driver.core.TransportException: [/*.*.*.*:9042] Unexpected exception triggered (com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042)
    at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:727)
    at com.datastax.shaded.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at com.datastax.shaded.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at com.datastax.shaded.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
    at com.datastax.shaded.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at com.datastax.shaded.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at com.datastax.shaded.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:141)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:83)
    at com.datastax.shaded.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at com.datastax.shaded.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: com.datastax.shaded.netty.channel.ConnectTimeoutException: connection timed out: /*.*.*.*:9042
    at com.datastax.shaded.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(NioClientBoss.java:137)
    ... 8 more

Answer 1

It looks like you are hitting connection timeouts here. A few ideas of what could be happening.

1. Your connection timeout in SocketOptions is tuned too low. Are you configuring this explicitly? If not the default is 5000ms.
2. Your rpc_address on your cassandra nodes' cassandra.yaml is configured for a private IP interface that your client cannot access. I noticed the presence of a 10.x.x.x ip in your log (Failed reconnection to /10.0.1.18:9042). The driver uses rpc_address (or broadcast_rpc_address if rpc_address is 0.0.0.0) to connect to cassandra hosts. Is this a possibility?
3. Your C* node is overloaded. If this is happening I doubt this is the C* node as usually when this happens you get operation (read) timeouts instead as the OS hosting cassandra accepts connections but your C* java process doesn't respond to requests because it is experiencing high pause rates from frequent garbage collections. In any case I would check system.log on your cassandra nodes to see if there is anything that indicates hosts going up and down, GCInspector with long pause times, or other messages that could indicate problems.