Spark Serialization workaround for 3rd party implicit vals

170 Views Asked by At

I'm running a Spark job that does some Http calls along the line. I decided to use a library Sttp which requires an implicit Http backend, in my case HttpURLConnectionBackend(): an object apply which creates a new class instance, which is not serializable.

trait ApiUtils extends MediaTypes {

  implicit val backend: SttpBackend[Identity, Nothing, NothingT] = HttpURLConnectionBackend()

  def post = { ...

As a consequence, because my Spark job extends this trait, all methods from this trait inside the Spark job are not available on the executors.

As a fairly brainless workaround, I made the val a def, but what's the recommended way for sorting these things out?

The Spark job itself is a class, not an object. Reason for this is to mock/stub the mentioned Http backend inside the tests.

0

There are 0 best solutions below