value _1 is not a member of org.apache.spark.mllib.recommendation.Rating

Question

I have an application with spark MLlib-scala, I want to split my data on 3 parties: Training, test, validation. My code is the following:

 val training_RDD = Ratingfiles.filter(x => x._1 < 6)
      .values
      .cache()

val validation_RDD = Ratingfiles.filter(x => x._1 >= 6 && x._1 < 8)
      .values
      .cache()

when I compile my program with sbt compile, I have this error:

value _1 is not a member of org.apache.spark.mllib.recommendation.Rating

Spark-core: 1.4.1 Spark-MLlib:2.0.1 Scala version: 2.11.1 Sbt version: 0.13.12

Answer 1

As the compiler claims, org.apache.spark.mllib.recommendation.Rating does not have a member called _1 (you're probably confusing it with a Tuple, for which the members are _1, _2 etc.).

Rating has three members:

case class Rating @Since("0.8.0") (
  @Since("0.8.0") user: Int,
  @Since("0.8.0") product: Int,
  @Since("0.8.0") rating: Double) 

So - if you mean to be filtering by user, simply access that member instead of _1:

val training_RDD = Ratingfiles.filter(x => x.user < 6)
  .cache()

val validation_RDD = Ratingfiles.filter(x => x.user >= 6 && x.user < 8)
  .cache()

Answer 2

Spark Rating class have 3 attributes (since spark 0.8.0) :

• user
• product
• rating

If you want to get the first value, you need to invoke user(); the second value product(); the third value rating()

sources : https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/mllib/recommendation/Rating.html

https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala