I am using Cassandra bulkloader to upload data to a cluster (the node used to upload data isn't part of the cluster). I have observed that in rare cases, the bulkload process hangs indefinely durin g data upload. The hang is not predictable, the same data upload succeeds if run next time.
Cluster details: 20 node Cassandra 2.1.11 EC2 cluster. Bulkloader node: 2.1.2 Cassandra
Following are the client and server logs for one such hang
Client logs:
016-01-16 10:40:20,466 INFO [STREAM-OUT-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.hadoop.hdfs.DFSClient: Seek offset: 141643669
2016-01-16 10:40:20,525 INFO [STREAM-OUT-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.hadoop.hdfs.DFSClient: Seek offset: 152129429
2016-01-16 10:40:21,282 INFO [STREAM-IN-ip-172-31-32-55.us-west-2.compute.internal./172.31.32.55] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-32-55.us-west-2.compute.internal./172.31.32.55 is complete
2016-01-16 10:40:21,368 ERROR [STREAM-IN-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.cassandra.streaming.StreamSession: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Streaming error occurred
java.lang.NullPointerException
at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:547)
at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:429)
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:251)
at java.lang.Thread.run(Thread.java:744)
2016-01-16 10:40:21,369 INFO [STREAM-IN-ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-44-254.us-west-2.compute.internal./172.31.44.254 is complete
2016-01-16 10:40:21,780 INFO [STREAM-IN-ip-172-31-32-51.us-west-2.compute.internal./172.31.32.51] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-32-51.us-west-2.compute.internal./172.31.32.51 is complete
2016-01-16 10:40:31,841 INFO [STREAM-IN-ip-172-31-32-53.us-west-2.compute.internal./172.31.32.53] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-32-53.us-west-2.compute.internal./172.31.32.53 is complete
2016-01-16 10:40:34,527 INFO [STREAM-IN-ip-172-31-44-255.us-west-2.compute.internal./172.31.44.255] org.apache.cassandra.streaming.StreamResultFuture: Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with ip-172-31-44-255.us-west-2.compute.internal./172.31.44.255 is complete
Server logs (Node for which stream failed):
INFO [STREAM-INIT-/172.31.37.149:51647] 2016-01-16 10:40:21,360 StreamResultFuture.java:109 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff ID#0 Creating new streaming plan for Bulk Load
INFO [STREAM-INIT-/172.31.37.149:51647] 2016-01-16 10:40:21,360 StreamResultFuture.java:116 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff, ID#0 Received streaming plan for Bulk Load
INFO [STREAM-INIT-/172.31.37.149:51652] 2016-01-16 10:40:21,362 StreamResultFuture.java:116 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff, ID#0 Received streaming plan for Bulk Load
INFO [STREAM-IN-/172.31.37.148] 2016-01-16 10:40:21,368 StreamResultFuture.java:180 - Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Session with /172.31.37.148 is complete
WARN [STREAM-IN-/172.31.37.148] 2016-01-16 10:40:21,368 StreamResultFuture.java:207 - Stream #49f7ad10-bc67-11e5-a004-e70ac8f0e359 Stream failed <<<<<<<<<<<<< Last entry in log for the stream id
INFO [STREAM-IN-/172.31.37.149] 2016-01-16 10:40:21,407 StreamResultFuture.java:166 - Stream #6cbdf570-bc67-11e5-b829-cf763a9770ff ID#0 Prepare completed. Receiving 1 files(29350007 bytes), sending 0 files(0 bytes)
INFO [StreamReceiveTask:230] 2016-01-16 10:40:22,159 StreamResultFuture.java:180 - Stream #507b0100-bc67-11e5-8fa7-e70ac8f0e359 Session with /172.31.37.148 is complete
INFO [StreamReceiveTask:230] 2016-01-16 10:40:22,159 StreamResultFuture.java:212 - Stream #507b0100-bc67-11e5-8fa7-e70ac8f0e359 All sessions completed
INFO [StreamReceiveTask:242] 2016-01-16 10:40:22,204 StreamResultFuture.java:180 - Stream #56ddad90-bc67-11e5-ae72-079f597e5002 Session with /172.31.37.150 is complete
INFO [StreamReceiveTask:242] 2016-01-16 10:40:22,205 StreamResultFuture.java:212 - Stream #56ddad90-bc67-11e5-ae72-079f597e5002 All sessions completed