I am new to Flink and Elastic Search integration. I have a scenario where i have to load history data(approx 1TB) from an old elastic search cluster(5.6) to new cluster(6.8). I have to do some data filtering and modification during the migration. Thinking about using a flink batch job with flink-es-sink operator.
But since there is no flink-es-source operator currently available , whats the best way to source the data into my flink pipeline. I have couple of options to do this.
- Write a flatmap function/process function and get the record
- Use some open source 3rd party libraries for connecting flink to ES. But dont want to take risk because dont know how these programs performs
But not sure which is the best way, since the data size is huge i might have to parallelize the source operator .
Please suggest few options if any of you have come across this scenario . Thanks in advance