How to handle Cassandra lost connections?

Question

I am fairly new to Cassandra Development using the DataStax C++ driver. I am currently using Cassandra 3.9 and the DataStax C++ driver 2.6. In my C++ application, I am attempting to catch situations where the driver has lost it's connection to Cassandra and then have the application re-attempt to make a connection to Cassandra. I am doing this as follows:

1. Catch the lost connection before doing a query by checking that cass_future_error_code(connection) != CASS_OK. If the result is true, then I know that the connection is lost.
2. I close the session, free the session, and free the cluster to release resources.
3. I recreate the resources and attempt to reconnect to Cassandra.

This approach would be repeated in a separate thread until the connection is re-established. I have tested this by shutting down the Cassandra Server. I had expected that the application would continue to retry to establish a connection until the Cassandra Server could be restarted and a connection was made. However, what happened was that the driver threw the following error and my application terminated:

1503345418.300 [WARN] (src/control_connection.cpp:267:virtual void cass::ControlConnection::on_close(cass::Connection*)): Lost control connection to host xx.xx.xx.xx with the following error: Read error 'connection reset by peer'
1503345418.300 [ERROR] (src/pool.cpp:384:virtual void cass::Pool::on_close(cass::Connection*)): Closing established connection pool to host xx.xx.xx.xx because of the following error: Read error 'connection reset by peer'
Error: Request timed out

Is there a better way to handle lost connections to Cassandra and implement a connection retry? Any examples or pointers to an example would be greatly appreciated.

Answer 1

The DataStax C/C++ driver automatically handles reconnections. The cass_cluster_set_reconnect_wait_time(Cluster*, time_in_ms) can be used to throttle the reconnect attempts (defaults to 2 seconds); https://github.com/datastax/cpp-driver/blob/2.6.0/include/cassandra.h#L1080-L1092. When the driver detects that a node has gone down it will automatically attempt a reconnect.