Amazon Keyspaces writing at high rate gives query timeout out after PT2S

Question

I am comparing Amazon CloudSearch and Amazon Keyspaces (managed Apache Cassandra-compatible database service) for a particular use case. I receive a stream of messages from an Apache Kafka topic. I need to upload and store those messages somewhere and later query them.

In CloudSearch, I could upload batches that included 12000 records in each. About 30000 records could be uploaded per minute without an issue.

With Keyspaces, I need to achieve at least the same write rate. I put the messages from the Kafka topic into a blocking queue in Java. Then a separate thread takes each message from the queue and writes them to the Keyspaces. I read about Batch Statements in CQL but since my messages are in different partitions, I don't think they are a good option.

My data uploading thread:

public void run() {
    while (true) {
        JsonNode record = null;
        try {
            record = queue.take();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        if (record != null) {
            awsKeyspacesService.uploadRecord(record, tableName);
            try {
                Thread.sleep(100);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

public void uploadRecord(JsonNode record, String table) {
    InsertInto insertInto = insertInto(keyspacesWriterConfig.getKeyspaceName(), table);
    SimpleStatement simpleStatement = insertInto
            .json(record.toString())
            .build()
            .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM);

    CompletionStage<AsyncResultSet> completionStage = session.executeAsync(simpleStatement);
    completionStage.thenAccept(result -> {
        logger.info("Wrote {} to {}", record.get("ID"), table);
    });
    completionStage.exceptionally(throwable -> {
        logger.warn("Error {} to {}: {}", record.get("ID"), table, throwable.getMessage());
        return null;
    });
}

With Thread.sleep(100), only 600 writes happen per minute. If I reduce the sleep, I get com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S.

Is there something I can do to achieve the same write rate as I could with CloudSearch?

I found this related question: Cassandra write query timeout out after PT2S

According to the answer given, I could be overloading the cluster.

I calculated the size of a row by referring to Calculating row size in Amazon Keyspaces. It was 322 bytes. In Quotas for Amazon Keyspaces, Max write throughput per second is 40,000 WRU. One WRU represents one write for a row up to 1 KB in size. Therefore it seems like it should be possible to achieve the rate.

Answer 1

I would suggest increasing observability into Keyspaces metrics to see if you are hitting any service capacity quotas. You can use the following cloudwatch repository to launch available metrics for table or keyspace. https://github.com/aws-samples/amazon-keyspaces-cloudwatch-cloudformation-templates

I have found in some cases the driver throws this error when creating too much asynchronous work. Try adding a semphore to control the number of asyc request that are running concurrently.