How to get number of rows written in spark 2.3 using JAVA?

Question

I know we can use the use count(). But I'm trying to capture the count using sparkListener. But I'm failing to write a proper java code for the same. I've tried following the exact approach given in this How to implement custom job listener/tracker in Spark? But I'm not able to reproduce it in JAVA. This is something that I've tried.

sparkContext.sparkContext().addSparkListener(new SparkListener()
                {
                    @Override
                    private void onTaskEnd(SparkListenerTaskEnd taskEnd){
                        taskEnd.taskInfo().accumulables().name
                    }
                });

But it throws a lot of errors related to override and name.Any help is appreciated. Thanks.

Answer 1

Cannot reduce the visibility of the inherited method from SparkListener

Change override method private to public code should work

sparkContext.sparkContext().addSparkListener(new SparkListener() {
            @Override
            public void onTaskEnd(SparkListenerTaskEnd taskEnd){
                    System.out.println(taskEnd.taskInfo().accumulables());
            }
        });