I have a question about the Ingress Module of Flink Stateful Functions. I want to:
- use the Delta Lake as ingress instead of Kakfa
- leverage knative or KEDA to automatically scale in/out based on the message size. This approach works with the Kafka ingress: non workload (Scale in Flink Stateful Functions) when there's no new message in Kafka, and scale out to ensure performance in case of a data spike in Kafka
Although I can change my system to: delta lake -> send to Kafka -> Flink Kafka Ingress, I still want to see if there're any methods to use Delta Lake directly, which is: Delta Lake -> Flink Delta Lake Ingress(not exist yet)
My questions are:
- Does Flink team have any plan for the DeltaLake Ingress? If not, what're my options here?
- I'm willing to try to develop one and contribute to Flink. Some guidelines are highly appreciated
- I see there's Flink Stateful Funtion's "embeded module", I guess it would work with Delta Lake, but I'm not sure if it would support the auto-scaling with KEDA/Knative
Thank you!