How to calculate network latency between operators in Apache Apex

Question

Is there a way, using which we can predict the N/W latency between two different operators in Apache Apex.

In a situation where the data that is going to be used in an operator, resides on a different machine altogether; this will add to the latency of the operator.

How can this be calculated?

Answer 1

You may want to look at a couple of blog articles:

https://www.datatorrent.com/blog/latency-calculation-in-apache-apex/ https://www.datatorrent.com/blog/blog-apex-performance-benchmark/

Answer 2

Assuming your tuples are strings and that the clocks on your cluster nodes are synchronized, you can append a timestamp to each tuple in the sending operator. Then, in the receiving operator, you can strip out the timestamp and compare it to the current time. You can, of course, suitably adapt this approach for other types. If averaged over a suitably large number of tuples, it should give you a good approximation of the network latency.