Stress testing Cassandra from C# consumes all network resources and fails after ~100K inserts

Question

We are currently evaluating a Windows Cassandra deployment and I have managed to write a pretty straight-forward stress test. The test attempts to insert 1 million rows with various data into a single local node (to/from localhost, no replication, only one node on the cluster).

This test works fine for the first 100k rows (or so, it does NOT always block on exactly the same row). The inserted rows can be read from the database correctly and everything works fine. Then the test seems to consume every available sockets/buffer/some network resource on the machine and the test fails to connect to Cassandra and stops inserting rows. Even refreshing a completely independent web page from a browser fails for a little while.

I first accused the test, but killing it and restarting it does not work. I need to wait about one minute for new connections to be available (timeout?). Then I may resume the test and keep pumping data. I wrote the test using both the Fluent Cassandra and Cassandra-Sharp clients with the same outcome.

Is there some setting that I need to set in Cassandra or in the C# clients like a maximum number of concurrent connections?

Alternatively, does anyone have a link to the sources of a working stress test which inserts a few million rows from C# so I may compare it with my test and figure out my mistake?

Answer 1

I used the Thrift API directly instead of the Cassandra Sharp or Fluent Cassandra APIs. It showed that this code works.

socket = new TSocket("127.0.0.1", 9160);
socket.Open();
client = new Cassandra.Client(new TBinaryProtocol(new TFramedTransport(socket)));
client.set_keyspace("Test");

foreach (Data data in myData) {
    Write(client, data);
}

socket.Close();

While the following code eventually reproduces the problem the higher level APIs encounter on "socket.Open()".

foreach (Data data in myData) {
    socket = new TSocket("127.0.0.1", 9160);
    socket.Open();
    client = new Cassandra.Client(new TBinaryProtocol(new TFramedTransport(socket)));
    client.set_keyspace("Test");

    Write(client, data);

    socket.Close();
}

The socket.Close() method does not release all of the socket's resources which is probably a bug.

Answer 2

My guess is that you aren't leaving the socket connection open. You can leave the connection open in FluentCassandra by wrapping the context in a session. If you don't do this, each request executed is on a new socket.