Zookeeper Watches Missed Events

Question

In Zookeeper, How do we handle "missing of data change" events between receiving the call back from previous watch and resetting the new watch. I am try to look at many solutions available like Apache curator and other suggestions in internet.But i am still not clear.

How do we ensure that we don't miss any events or is there any other fool proof way of ensuring that we don't miss any state changes in zookeeper(especially node data changes)?? I understand regular polling is one way. But that's costly.

Answer 1

First off, I can't recommend using Apache Curator enough. It has some great wrappers to make things super convenient and more robust. It handles a lot of the stuff you might otherwise have to code up yourself (and stuff that is hard to get right).

Second, see this tech note in the curator docs:

https://cwiki.apache.org/confluence/display/CURATOR/TN1

Main takeaway here is that any code you write to respond to a watch notification should return as rapidly as possible and should never block (on i/o or waiting for a response from a subsequent zk request, &c).

I would therefore recommend you (1) spawn a thread to handle the code that the watch kicks off and (2) as you would in any distributed/concurrent scenario, verify the data as one of the first things that code in that new thread does. It has some overhead, but it's better than polling.

Regular polling should never be in play with zookeeper - that is what the watches are for. You know when to check whether the data are changed. You check when the data watch triggers exactly to capture the kinds of subsequent updates you're concerned about.

Answer 2

I do not think there is a way to never miss an update/event.

However, since receiving the WatchedEvent object, the client code has to read the latest value. It is not passed through the event object. While doing so, either a new or the same Watcher instance is passed with the read-method invocation signing up for future event notifications. This means your client will always get the latest value, and if anything changed after this method invocation, a new event will be fired.

So although events might be missed between the receipt of the event and the operation to retrieve the latest and greatest value, your client will still get the latest value. He will never not get the latest value. Changes or values in-between is generally speaking not of interest and if they are, then you should probably skip ZooKeeper and look for another communication mechanism.

Massive disclaimer: I am completely new to ZooKeeper and this answer reflects only knowledge gained through reading and googling. Which sort of explains why I dare not add example code to this answer. LOL.