How to handle Coders for custom case class

178 Views Asked by At

I try to get a simple SCIO code running.

case class Foo(first: String, second: String, third: String)

Trying to use Foo in a SCollection it leads to an error:

Cannot find an implicit Coder instance for type:

  >> (String, my.Foo)
...
<redacted prose>
...

There is a lot written in the error message. I cannot seem to grasp, though how all that text provides any information to what I am supposed to do around Coders to solve this problem.

Can anyone shed some light how one would solve this.

Bonus points for not using implicits.

1

There are 1 best solutions below

0
Jayadeep Jayaraman On

Here is a minimalistic example

case class DemoEvents (reverse_geo_code_failed:String
                         ,eventType:String
                         ,city:String
                         ,area:String
                        )

class DoFnExample extends DoFn [String, DemoEvents]{
    // `@ProcessElement` is called once per element.
    @ProcessElement
    private[example] def processElement(c: DoFn[String, DemoEvents]#ProcessContext): Unit ={
      //val t = DemoEvents("BNG1111","pickup","ORR","1111")
      implicit val formats: DefaultFormats.type = DefaultFormats
      c.output(parse(c.element()).extract[DemoEvents])

    }
  }

val eventStream: SCollection[String] = sc.pubsubSubscription[String](args("pubsub-subscription-name"))
      .withFixedWindows(Duration.standardSeconds(120L))

val events: SCollection[DemoEvents] = eventStream.applyTransform(ParDo.of(new DoFnExample()))