Azure Eventhub, pausing capture looses events when restarted

Question

Mysql DB with Java front end sending compressed and encrypted messaged to eventhub. All fine here. EventHub Capture is every 5 mins sending Avro with all messages to Storage account.

For maintenance and upgrades it is sometimes necessary to pause Capture, the issue we appear to have is when we switch on Capture we loose the first set of messages/ those that are stored on the Eventhub between capture off and on. The Avros in size do not exceed the file size to be transmitted and the messages are well within purge date/time of keeping messages.

This obviously gives us an issue where we miss critical messages. Not sure what additional info is required to investigate but some pointers would be very helpful please.

Thanks in advance for any assistance.

setting capture to enabled all the messages NOT processed / captured.

Answer 1

In my opinion, I don't know if Azure Event Hubs has native version control functionality. otherwise we could consider using the version control functionality in Azure Event Hubs. This should allow you to create a new Hub event with a new capture instance, pause the old capture, then re-enable the new capture. This ensures that messages are not lost during the transition.

Answer 2

Event Hubs Capture does not resume capturing events from the last known position when capture is re-enabled. Instead, it starts capturing events from the current time. This means that any events that occurred while capture was disabled will not be captured. If you want to capture all the events in your event hub, you should avoid disabling and re-enabling capture. Alternatively, you can use another consumer to read the events from the event hub and store them in a different location.