How to fine-tune QuestDB to ingest into 1,000 tables or more?

Question

I'm trying to ingest data into 1,000 tables in QuestDB. But my machine runs out of free disk space quickly as soon as I start my InfluxDB Line Protocol client. Occasionally, I'm also getting Linux' OOM killer killing database process due to high memory usage.

Is it possible to configure QuestDB for simultaneous ingestion into lots of tables?

Answer 1

Yes, in case of ingestion into many smaller tables, it's possible to reduce QuestDB's disk and memory footprint. You put the following into your instance's server.conf file:

cairo.o3.column.memory.size=256K
cairo.writer.data.append.page.size=256K
cairo.wal.writer.data.append.page.size=256K
cairo.writer.data.index.key.append.page.size=256K
cairo.writer.data.index.value.append.page.size=256K

The first setting limits the default buffer size used to handle Out-Of-Order (O3) (think, unsorted) data. When there are a lot of tables (and, thus, columns), these buffers may require a significant portion of the machine's memory. Next, we're lowering down *.append.page.size settings: these define sizes in which the database grows column and index files when the current size is insufficient to append a new row. The default values of these settings are different, but they're in the 8-16 MB range.

So, by setting all of these to 256 KB, we're limiting the overall RAM and disk footprint.