Kafka streams RocksDB as an EventStore?

Question

There are alot of articles about kafka and event sourcing. Most of the articles are about that kafka is not realy usefull when doing eventsourcing, because you cannot query the events just for a given aggregate id. If you store the events in a topic, then yes this is true. This is because we need to read all the events and skip those that are not relevant. But what about storing the events in rocksdb? Now, we can actually query all the events just for the given aggregate by using the aggregate id as prefix and do a range query in rocksdb.

Is this a good approach? I know that this will use large state and can be problematic when a rebalance occurs. But again, maybe static membership in kafka will help with this.

Answer 1

Kafka Streams's default disk-based StateStore is RocksDB, so yes, this is a perfectly valid approach.

You'd query the store via Kafka Streams APIs, not RocksDB directly, however.

Now, we can actually query all the events just for the given aggregate by using the aggregate id as prefix

Unclear what you mean by prefix. The stores are built exclusively by Kafka record keys, not by prefixed values. However, as linked in the comments, the store does support prefix scanning, but I assume that'd be the prefix of the kafka record keys

this will use large state and can be problematic

You can refer the Memory Management page on handling state and what to tune for handling its size.

Answer 2

Kafka Streams and RockDB is really good solution for a quick startup to understand the concepts but I am not sure it is good idea in the long term for the production.

My personal experience with RockDB was not that brilliant in production, if you plan to use a Key/Value database in production Apache Cassandra seems to be a much better solution.

But you are also right, querying things only over Primary Key is not that flexible, so implementing CQRS does make much more sense, so you get much more flexibility for Query Side.

As a person that walked the path you plan the walk, you can find my proposed solution in my blog :)