So, I am very very new to all the Apache Frameworks I am trying to use. I want your suggestions on a couple of workflow design for an IoT streaming application:
- As we have NiFi connectors available for Flink, and we can easily use Beam abstraction over Flink. Can I use NiFi as the dataflow tool to drive data from MiNiFi to Flink Cluster (here store it in-memory or something) and then use Beam Pipeline to process the data further.
- Is there any NiFi connector for the beam ? If not can we do so? So, we directly stream data from NiFi to the Beam job (running on a Flink Cluster)
I am still in the early design phase, it would be great if we can discuss possible workarounds. Let me know if you need any other details.