Datomic: robust settings to avoid timeouts and out-of-memory errors

Question

I'm running the same datomic-backed application on a variety of architectures with varying amounts of memory (1GB - 16GB). When I do bulk imports of data I frequently run into timeouts or out-of-memory errors.

After looking at the documentation I happened upon this helpful document (and this one) which seem to outline best practices to obtain good performance under heavy imports.

I'm not as interested in performance as I am in making imports "just work." This leads to my main question:

What is the minimum complexity configuration to ensure that an arbitrarily large import process terminates on a given machine?

I understand that this configuration may be a function of my available memory, that's fine. I also understand that it may not be maximally performant; that's also fine. But I do need know that it will terminate.

Answer 1

Data Distribution

I think the critical pieces of information missing from your question are the type of data and its distribution and the available metrics for your system during the bulk imports. Why?

Datomic's transaction rate is limited by the cost of background indexing jobs and the cost of that indexing is a function of the distribution of new values and the size of your database.

What this means is that, for example, if you have indexed attributes (i.e. :db/index) and, as your bulk import goes through, the distribution of those attribute values is random, you'll put a lot of pressure in the indexing jobs as it rewrites an ever increasing number of segments. As your database size grows, the indexing will dominate the work of the transactor and won't be able to catch up.

Transactor Memory

As described in the docs, the more memory you can give to object-cache-max, the better. This is especially important if your data has a lot uniqueness constraints (i.e. db/unique) since that will prevent the transactor from fetching some storage segments multiple times.

Depending on your data distribution, increasing the memory-index-threshold and memory-index-max settings may let your imports run longer... until the indexing job can't keep up. This seems to be what it's happening to you.

Recommendations

Try reducing the memory-index-threshold and memory-index-max settings. That may seem counterintuitive but you'll have much better chances to have any import complete (of course they'll take more time, but you could almost guarantee they will finish). The key is to make the transactor throttle your (peer) requests before it's not able to catch up with the indexing jobs.