Dynamic Filter with Spark Structured Streaming

Question

I am working on my Spark Streaming project with the goal of creating a simple app to notify users when a data stream meets a condition (e.g., sending a notification when a stock price > x).

df = spark.readStream...\
          .filter(col > x)

However, I am unsure how to address the issue of allowing users to change the condition (such as changing the value of x) while the data is still streaming without generating redundant messages (messages for the old price or condition).

Could you please help me with some solutions using Spark Structured Streaming, preferably in Python, to solve this problem?

Thank you so much!

Answer 1

You have to restart the StreamingQuery to reflect changes in x. StructuredStreaming won't process "old" messages again if you have configured the checkpointLocation, see Checkpointing.