How to replay Kinesis records with KCL C#

Question

I want to Replay Kinesis Records by a given Timespan, (or a start time)

use case:

on application failure, some stream records might be considered processed, but not actually being finalized. i.e. not being saved in local data store. so on system recovery, we would like to start processing records 30 minutes earlier (or any given number) of current checkpoint.

I came across this https://github.com/scopely/kinesis-vcr it persists Kinesis records into S3, then replay the events from S3

I am wondering if it is possible to replay the records without s3, since Kinesis Keeps records for 24 hours. and 24 hours is long enough.

Answer 1

You need to specify a ShardIteratorType of TIMESTAMP so.

I think that's probably the wrong way to go about it though. Best to keep a sequence number of successfully processed records and simply have your app use that to process the next records (start using a ShardIteratorType of AT_SEQUENCE_NUMBER). If you build it programatically like that you take the guess work out of going back x time period.

Answer 2

Time-based iterators make your streams easily replay-able by your processing applications. Using time-based iterators, you can read records from a position in the stream corresponding to a timestamp.

This gives you the ability to go back in time and replay the stream for any time window within the default 24-hour or the extended 7-day data retention period. Java Streams API has something called GetShardIterator, you can try to find something similar to that in C#