Spark MLLIB error: java.lang.NoSuchMethodError: org.apache.spark.rdd.RDD.treeAggregate

826 Views Asked by At

I am trying to run the Linear Regression example in the Spark job. I got the following error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.rdd.RDD.treeAggregate$default$4(Ljava/lang/Object;)I
    at org.apache.spark.mllib.optimization.GradientDescent$$anonfun$runMiniBatchSGD$1.apply$mcVI$sp(GradientDescent.scala:189)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at org.apache.spark.mllib.optimization.GradientDescent$.runMiniBatchSGD(GradientDescent.scala:184)
    at org.apache.spark.mllib.optimization.GradientDescent.optimize(GradientDescent.scala:107)
    at org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(GeneralizedLinearAlgorithm.scala:267)
    at org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(GeneralizedLinearAlgorithm.scala:190)
    at com.myproject.sample.LinearRegression$.run(LinearRegressionExample.scala:105)
    at com.myproject.sample.LinearRegression$$anonfun$main$1.apply(LinearRegressionExample.scala:67)
    at com.myproject.sample.LinearRegression$$anonfun$main$1.apply(LinearRegressionExample.scala:66)
    at scala.Option.map(Option.scala:145)
    at com.myproject.sample.LinearRegression$.main(LinearRegressionExample.scala:66)
    at com.myproject.sample.LinearRegression.main(LinearRegressionExample.scala)

Here is my dependencies in pom:

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.10</artifactId>
        <version>1.3.1</version>
    </dependency>

    <dependency> 
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.2.0</version>
        <exclusions>
         :
         :
    </dependency>

Why did I get java.lang.NoSuchMethodError: org.apache.spark.rdd.RDD.treeAggregate ? Did I use the wrong dependency? Thanks!

0

There are 0 best solutions below