Delta Lake as ingress for Flink Stateful Functions

14 Views Asked by At

I have a question about the Ingress Module of Flink Stateful Functions. I want to:

  • use the Delta Lake as ingress instead of Kakfa
  • leverage knative or KEDA to automatically scale in/out based on the message size. This approach works with the Kafka ingress: non workload (Scale in Flink Stateful Functions) when there's no new message in Kafka, and scale out to ensure performance in case of a data spike in Kafka

Although I can change my system to: delta lake -> send to Kafka -> Flink Kafka Ingress, I still want to see if there're any methods to use Delta Lake directly, which is: Delta Lake -> Flink Delta Lake Ingress(not exist yet)

My questions are:

  1. Does Flink team have any plan for the DeltaLake Ingress? If not, what're my options here?
  2. I'm willing to try to develop one and contribute to Flink. Some guidelines are highly appreciated
  3. I see there's Flink Stateful Funtion's "embeded module", I guess it would work with Delta Lake, but I'm not sure if it would support the auto-scaling with KEDA/Knative

Thank you!

0

There are 0 best solutions below