Hazelcast NearCache doesn't have the expected effect on performance

Question

We have an application running on 1 or 2 nodes, depending on the environment, with a shared cache based on Hazelcast.

One of the requests on the application triggers about 1000 requests on this cache (all cache hits).

In a single node configuration this works nicely. Specifically each request takes less then 10ms.

But if we use 2 nodes, each cache request takes about 20-200ms. We thought this would be due to Hazelcast getting the data from the remote node, which of course involves network traffic. So we configured it to use a NearCache, which to our understanding should result in about the same access speed as a single local cache. But it doesn't, it doesn't seem to have an effect on the performance at all.

So now I'm wondering:

• How can I check if the NearCache configuration actually works?
• How can I get read performance close to a local cache, but updates (asynchronously) communicated to the/all remote caches, which I thought I would get by configuring a NearCache?

We initialize our cache with the following configuration:

HazelcastConfiguration.getOrCreateCache(
                cacheName,
                new CacheConfig<K, CR>()
                        .setNearCacheConfig(new NearCacheConfig())
                        .setExpiryPolicyFactory(
                                HazelcastConfiguration.createExpiryPolicyFactory(expiryAfterModification)),
                cacheManager
        );

Answer 1

It depends on a lot of things, but 20/200ms is a lot. When we run a 4 node cluster benchmark (4 dual socket xeon machine) on 1 standard 1 GbE network, 99 percentile is far below 1 ms. We are talking here without nearcache.

Near cache should give very good performance because if the data is locally cached, then the call is local + the whole internal infrastructure for operation execution is skipped (at least for the IMap and the near cache; not completely sure about the Cache.

Can you try hitting the same set of key(s)? Just to make sure you are hitting the near cache.

What are you using as key and value? Are they large or serialization unfriendly?

I you are still running into problems; quickly try to IMap and enable near cache and check if you see big differences.