Why is my pipeline throwing a "fetchMessages request failed with invalid offset error"?

Question

I am trying to write a streaming pipeline and received this error.

Here is my spec: -

--cpu-parallelism 8
--memory-parallelism 8

Please let me know if this makes sense.

Answer 1

There are two known scenarios for such behavior -

• If the Pipeline was paused for longer than the retention period of one of the input streams OR
• If the Pipeline is processing data slower than the input streams are receiving data. In such cases, at some point, the current data being processed by Pipeline will get dropped by the stream layer as it crosses the retention period threshold.