We are trying to measure latency of message writes in Infosphere CDC- Confluent kafka setup. We have dummy table and dummy transactions will be performed.
Dataflow- MF DB2 table --> IIDR --> Kafka topic
(For Kafka,the producer is iidr)
What could be the right way to find out write latency of sample transactions without any enterprise monitoring tool? Can we check the logs for writes and consider each transaction timestamp and compare with cdc transaction logs to find out the write latency?
PS- I have jconsole configured to collect kafka metrics. But how can we find write latency of a particular CDC stream to kafka topic?
Thanks!
(FYI: I work for IBM and for the IDR CDC Product Specifically)
The integrated KCOPs (Kafka Custom Operation Processors) in the IDR Replication product do not supply a timestamp argument when creating the Kafka producer records. This is also the case if you use the default behaviour without a KCOP.
As such Kafka will generate a timestamp for each producer record when it receives it and associate it with the record. This timestamp value is included with the consumer record when the message is read.
This can be compared with a time generated from the source, by subtraction. There is a timestamp journal control field which can be replicated with the row data. On some sources, Db2 LUW for instance, it represents the commit time of the transaction. On some represents the time that particular row was added to the log.
https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.mcadminguide.doc/refs/recordmodificationtime.html
Generally I would argue that latency is best considered from the time the commit actually takes effect, but it depends on what you are looking to understand.
I believe on DB2 Z, each one is the time of the operation's log entry, you could take a rough approximation by using the timestamp entry of the last operation in the transaction. This ignores the amount of delay between the writing of the last transaction and the actual commit, but typically these are quite close.
IDR does offer requests for feature enhancement, so please feel free to reach out through either support or this if you'd like to see an enhancement.
Regards,