How to shade packages inside a fat jar depdency

255 Views Asked by At

I've an SBT project that depends on

"com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop3-2.2.2"

which is bringing a recent version of google-api-services-storage.

I've also another dependency to Sparkling Water which is a fat jar that seems to bring a different version of google-api-services-storage packaged with the rest of this library classes inside that fat jar.

Now whenever I try to use gcs-connector it fails with this error:

Caused by: java.lang.NoSuchMethodError: com.google.api.services.storage.Storage$Objects$List.setIncludeTrailingDelimiter(Ljava/lang/Boolean;)Lcom/google/api/services/storage/Storage$Objects$List;
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.createListRequest(GoogleCloudStorageImpl.java:1401)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.listStorageObjectsAndPrefixes(GoogleCloudStorageImpl.java:1272)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.listObjectInfo(GoogleCloudStorageImpl.java:1443)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.lambda$getFileInfoInternal$8(GoogleCloudStorageFileSystem.java:1059)

It seems the class loader is picking up the classes from inside that fat jar.

In this scenario how can I shade the classes inside the fat jar? I tried the following but without success (see https://github.com/h2oai/sparkling-water/issues/2643)

settings = Seq(
  ...,
  libraryDependencies ++= Seq(...),
  assemblyShadeRules in assembly ++= Seq(
    ShadeRule.rename("com.google.api.services.storage.**" -> "shade.@1")
      .inLibrary("the fat jar group" % "the fat jar linbary" % "the fat jar version")
  )
)

Update: after shading google dependency in the original I did run into another dependency issue:

Caused by: java.lang.NoSuchMethodError: io.opencensus.trace.Span.addMessageEvent(Lio/opencensus/trace/MessageEvent;)V
    at com.google.api.client.http.OpenCensusUtils.recordMessageEvent(OpenCensusUtils.java:222)
    at com.google.api.client.http.OpenCensusUtils.recordSentMessageEvent(OpenCensusUtils.java:190)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1010)
    at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:304)
    at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:324)
    at com.google.cloud.hadoop.util.CredentialFactory$GoogleCredentialWithRetry.executeRefreshToken(CredentialFactory.java:170)
    at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:470)
    at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:201)
0

There are 0 best solutions below