I want to understand, What role "spark.streaming.blockInterval"
plays in Spark Streaming DirectAPI, as per my understanding "spark.streaming.blockInterval"
is used for calculating partitions i.e. #partitions = (receivers x* batchInterval) /blockInterval
, but in DirectAPI spark streaming partitions is equal to no. of kafka partitions.
How "spark.streaming.blockInterval"
is used in DirectAPI ?
spark.streaming.blockInterval :
And KafkaUtils.createDirectStream() do not use receiver.