Warning that "unknown addresses are found in partition table"

Question

When using Hazelcast, I get warnings like:

Jun 21, 2015 11:10:15 AM com.hazelcast.partition.InternalPartitionService
WARNING: [192.168.0.18]:5701 [5a11] [3.4.2] Following unknown addresses are found\
in partition table sent from master[Address[192.168.0.9]:5701]. 
(Probably they have recently joined or left the cluster.) {
        Address[192.168.0.13]:5701 } 
Jun 21, 2015 11:10:29 AM com.hazelcast.partition.InternalPartitionService 
WARNING: [192.168.0.18]:5701 [5a11] [3.4.2] Following unknown addresses are found\
in partition table sent from master[Address[192.168.0.20]:5701]. 
(Probably they have recently joined or left the cluster.) {
        Address[192.168.0.11]:5701
        Address[192.168.0.17]:5701 }  
Warning: irregular exit, check log

What is the cause, and do I have to take actions to avoid these warnings?

---

Details:

These warning occur at the end of my distributed computations, and not for all instances. So it is very likely that some other instances have terminated and thus "recently left the cluster" when this warning occurs.

But why does an instance leaving cause an unknown address? Does this mean the instance x giving the warning somehow found out that instance y has left, and the master hasn't yet found out and sends the address of y to x, causing this warning?

Should I take actions to avoid this warning? Does it mean that y forgets some cleanup it is supposed to do at the end so that the master immediately finds out that y leaves the cluster? The only cleanup the instances are performing is shutdown() of their HazelcastInstance.

Is the irregular exit at the end of my log messages caused by the inconsistency in the partition table?

Answer 1

To achieve even data distribution, the whole key space is divided into a configurable number of partitions (271 by default). If you know the key and a number of partitions you can always tell to which partition a key belongs.

Even though Hazelcast is a peer-to-peer system and does not have a notion of main node, there is a master node which calculates the partition table that correlates a partition number to a node address that contains said partition.When cluster topology is changed the partition table needs to be recalculated.

I believe your problem is caused by non-graceful shutdown of the nodes.