I have data like below
val t=((1,List(1,2,3,4)),(2,List(1,2,3,4)),(3,List(1,2,3,4)),(4,List(1,2,3,4)))
and I want output like :
1--1
2--2
3--3
4--4
Can some body please help me here by using scala or spark core.
I have data like below
val t=((1,List(1,2,3,4)),(2,List(1,2,3,4)),(3,List(1,2,3,4)),(4,List(1,2,3,4)))
and I want output like :
1--1
2--2
3--3
4--4
Can some body please help me here by using scala or spark core.
Try this:
val spark = SparkSession.builder()
.appName("app_name")
.master("local[2]")
.getOrCreate()
import spark.implicits._
val df = spark.sparkContext.parallelize(Seq((1,List(1,2,3,4)),(2,List(1,2,3,4)),(3,List(1,2,3,4)),(4,List(1,2,3,4))))
df.map{
case (id, list) => s"${id}--${list(id - 1)}"
}.toDF().show()
Another way using RDD:
Input:
Output: