I have a problem with Spark Scala converting an Iterable (CompactBuffer) to individual pairs. I want to create a new RDD with key-value pairs of the ones in the CompactBuffer.
It looks like this:
CompactBuffer(Person2, Person5)
CompactBuffer(Person2, Person5, Person7)
CompactBuffer(Person1, Person5, Person11)
The CompactBuffers could obtain more persons than just 3. Basically what I want is a new RDD that has the individual combinations of the CompactBuffer like this (I also want to avoid identical key-values):
Array[
<Person2, Person5>
<Person5, Person2>
<Person2, Person7>
<Person7, Person2>
<Person5, Person7>
<Person7, Person5>
<Person1, Person5>
<Person5, Person1>
<Person1, Person11>
<Person11, Person1>
<Person5, Person11>
<Person11, Person5>]
Can someone help me?
Thank you in advance
Here's something that produces the pairs (and removes repeated ones). I couldn't work out how to use
CompactBuffer
so it usesArrayBuffer
, since the source for CompactBuffer says it's a more efficientArrayBuffer
. You may need to convert yourCompactBuffer
in theflatMap
to something that supports.combinations
.Output