implement custom partitioning with windowAll()

Question

I'm trying to implement custom partitioning with windowing in apache flink. Currently I have the following:

DataStream<Tuple2<String, Integer>> split = operatorAggregateStream
                .partitionCustom(new RoundRobin(), value->value.f0 )
                .windowAll(TumblingEventTimeWindows.of(Time.seconds(5)))
                .process(new MaxPartialWindowProcessFunction());

The issue is that I need windowing but using windowAll overrides my partitioning and creates a different partitioning.

my new solution is to forcefully implement myself the windowing but I feel that this is overkill. Is there any workaround?

Answer 1

The windowAll operation forces a parallelism of 1 (all records have to be sent to the same sub-task), since it operators on a non-keyed stream. This is why the partitioning you did previously isn't useful, as Flink will insert a rebalance that forces the stream parallelism to be 1.

You said "I need windowing", which means you need a keyed stream. You can do operations on partitioned records without windowing, but you won't be able to use timers to trigger window evaluations, as that requires a keyed stream.