Is faust[rocksdb] a mandatory extension to be installed?

Question

I am working on microservices architecture and using kafka to communicate b/n microservices. I am using python faust to implement producers and consumers.

Is it mandatory to install faust[rocksdb] extension? When I install it, fails with a

fatal error: rocksdb/slice.h: No such file or directory.

What could be the impact I could see in production if I am not using rocksdb? If the package is mandatory to install how can I solve the issue? I had tried many answers, but nothing seems to work.

Answer 1

RocksDB is not a required dependency of Faust. By default a Faust app saves changelogs to memory, which is in their documentation.

Some systems may have issues installing faust[rocksdb] but I've had success installing it inside a Docker container. I suggest reading this SO answer for more details since they're having the same issue with slice.h.

Be warned that the wheel on PyPi for python-rocksdb is outdated, you'll need to manually install a newer version from https://github.com/twmht/python-rocksdb. There are more features in its actively maintained fork (https://github.com/NightTsarina/python-rocksdb), but I only got that working inside an Ubuntu 22.04 container.

What could be the impact I could see in production if I am not using rocksdb?

If your Faust application needs to frequently restart and your changelogs have a lot of activity, then you'll have to wait a while to wait for your app to update your local state. This is also mentioned in the Faust documentation. This wait time could range from a few seconds to minutes depending on how frequently your tables change.

Update (08/11/2022): I have updated faust-streaming's fork of python-rocksdb, try this:

pip3 install faust-streaming-rocksdb

The C++ dependencies are bundled with this now and I'm able to install it onto Ubuntu 18.04 LTS.

Disclaimer: I'm a maintainer of the fork.