DEVHIDE
Login Or Sign up

sbt assembly command shows error: (redshiftConnector/*:assembly) deduplicate: different file contents found

1.2k Views Asked by At

sbt assembly is giving me errors i am trying to create a fat jar with this link https://github.com/sbt/sbt-assembly under the heading Exclude specific transitive deps they exclude some of the libs that are causing this issue but in my projects build.sbt file no dependency is there which is showing on sbt console . here is the stack trace 

[info] Including from cache: scala-library-2.11.7.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] Assembly up to date: /home/user/Downloads/ScalaProj/datasembly/target/scala-2.11/datasembly-assembly-0.1-SNAPSHOT.jar
[info] Including from cache: jackson-datatype-jsr310-2.5.4.jar
[info] Including from cache: scala-library-2.11.7.jar
[info] Including from cache: httpcore-4.3.2.jar
[info] Including from cache: postgresql-9.4-1201-jdbc41.jar
[info] Including from cache: commons-codec-1.6.jar
[info] Including from cache: slf4j-nop-1.6.4.jar
[info] Including from cache: amazon-kinesis-producer-0.10.1.jar
[info] Including from cache: joda-time-2.8.2.jar
[info] Including from cache: HikariCP-2.3.8.jar
[info] Including from cache: chill_2.11-0.7.0.jar
[info] Including from cache: chill-java-0.7.0.jar
[info] Including from cache: javassist-3.18.2-GA.jar
[info] Including from cache: kryo-2.21.jar
[info] Including from cache: guava-18.0.jar
[info] Including from cache: slick-pg_2.11-0.9.1.jar
[info] Including from cache: reflectasm-1.07-shaded.jar
[info] Including from cache: slf4j-api-1.7.12.jar
[info] Including from cache: asm-4.0.jar
[info] Including from cache: slick-pg_core_2.11-0.9.1.jar
[info] Including from cache: minlog-1.2.jar
[info] Including from cache: protobuf-java-2.6.1.jar
[info] Including from cache: slf4j-simple-1.7.12.jar
[info] Including from cache: objenesis-1.2.jar
[info] Including from cache: commons-io-2.4.jar
[info] Including from cache: flyway-core-3.2.1.jar
[info] Including from cache: commons-lang-2.6.jar
[info] Including from cache: aws-java-sdk-core-1.9.37.jar
[info] Including from cache: commons-logging-1.1.3.jar
[info] Including from cache: play-json_2.11-2.4.2.jar
[info] Including from cache: play-iteratees_2.11-2.4.2.jar
[info] Including from cache: httpclient-4.3.4.jar
[info] Including from cache: play-datacommons_2.11-2.4.2.jar
[info] Including from cache: joda-convert-1.7.jar
[info] Including from cache: scala-stm_2.11-0.7.jar
[info] Including from cache: scala-reflect-2.11.6.jar
[info] Including from cache: config-1.3.0.jar
[info] Including from cache: jackson-core-2.5.4.jar
[info] Including from cache: jackson-annotations-2.5.4.jar
[info] Including from cache: slick_2.11-3.0.0.jar
[info] Including from cache: play-functional_2.11-2.4.2.jar
[info] Including from cache: reactive-streams-1.0.0.jar
[info] Including from cache: jackson-databind-2.5.4.jar
[info] Including from cache: jackson-datatype-jdk8-2.5.4.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/DEPENDENCIES' with strategy 'discard'
[warn] Merging 'META-INF/INDEX.LIST' with strategy 'discard'
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/amazon-kinesis-producer/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/amazon-kinesis-producer/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/aws-java-sdk-core/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.amazonaws/aws-java-sdk-core/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.kryo/kryo/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.kryo/kryo/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.minlog/minlog/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.minlog/minlog/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.reflectasm/reflectasm/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.esotericsoftware.reflectasm/reflectasm/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jdk8/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jdk8/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.guava/guava/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.guava/guava/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.protobuf/protobuf-java/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.google.protobuf/protobuf-java/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.zaxxer/HikariCP/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/com.zaxxer/HikariCP/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-codec/commons-codec/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-codec/commons-codec/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-io/commons-io/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-io/commons-io/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-lang/commons-lang/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-lang/commons-lang/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-logging/commons-logging/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/commons-logging/commons-logging/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/joda-time/joda-time/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/joda-time/joda-time/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpclient/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpclient/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpcore/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.apache.httpcomponents/httpcore/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.flywaydb/flyway-core/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.flywaydb/flyway-core/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.javassist/javassist/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.javassist/javassist/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-nop/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-nop/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-simple/pom.properties' with strategy 'discard'
[warn] Merging 'META-INF/maven/org.slf4j/slf4j-simple/pom.xml' with strategy 'discard'
[warn] Merging 'META-INF/services/com.fasterxml.jackson.databind.Module' with strategy 'filterDistinctLines'
[error] 5 errors were encountered during merge
                                                                        [info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[info] Assembly up to date: /home/user/Downloads/ScalaProj/datasembly/galactus/target/scala-2.11/galactus-assembly-0.1.0.jar
[trace] Stack trace suppressed: run last redshiftConnector/*:assembly for the full output.
[error] (redshiftConnector/*:assembly) deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /home/user/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /home/user/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-nop/jars/slf4j-nop-1.6.4.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-simple/jars/slf4j-simple-1.7.12.jar:org/slf4j/impl/StaticLoggerBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-nop/jars/slf4j-nop-1.6.4.jar:org/slf4j/impl/StaticMDCBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-simple/jars/slf4j-simple-1.7.12.jar:org/slf4j/impl/StaticMDCBinder.class
[error] deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-nop/jars/slf4j-nop-1.6.4.jar:org/slf4j/impl/StaticMarkerBinder.class
[error] /home/user/.ivy2/cache/org.slf4j/slf4j-simple/jars/slf4j-simple-1.7.12.jar:org/slf4j/impl/StaticMarkerBinder.class
[error] Total time: 41 s, completed Oct 1, 2015 5:21:47 PM

here is my build.sbt files 

import com.amazonaws.services.s3.model.Region
import ohnosequences.sbt.SbtS3Resolver.autoImport._

name := "h"

lazy val commonSettings = Seq(
  organization := "datasembly",
  version := "0.1.0-SNAPSHOT",
  scalaVersion := "2.11.7",
  resolvers += "akka" at "http://repo.akka.io/snapshots",
  resolvers += Resolver.sonatypeRepo("public"),
  publishMavenStyle := false,
  s3region := Region.US_Standard,
  awsProfile := "maven",
  publishTo := {
    val prefix = if (isSnapshot.value) "snapshots" else "releases"
    Some(s3resolver.value("My "+prefix+" S3 bucket", s3(prefix+".mvn-repo.d")) withIvyPatterns)
  }
)

lazy val commonDepends = Seq(
  libraryDependencies ++= Seq(
    "com.github.scopt"            %%  "scopt"                     %   "3.3.0"
  )
)

lazy val clusterDepends = Seq(
  libraryDependencies ++= Seq(
    "com.typesafe.akka"       %%  "akka-cluster"      %   "2.4-M2",
    "com.twitter"             %%  "chill-all"         %   "0.7.0",
    "com.twitter"             %%  "chill-akka"        %   "0.7.0"
  )
)

lazy val root = (project in file(".")) aggregate(core, worker, master, slick, aws) settings(commonSettings: _*)

lazy val core = project settings(commonSettings: _*) settings(clusterDepends: _*)

lazy val worker = project dependsOn core settings(commonSettings: _*) settings(commonDepends: _*) settings(clusterDepends: _*) enablePlugins JavaAppPackaging enablePlugins UniversalPlugin

lazy val master = project dependsOn core settings(commonSettings: _*) settings(commonDepends: _*) settings(clusterDepends: _*) enablePlugins JavaAppPackaging

lazy val client = project dependsOn core settings(commonSettings: _*) settings(commonDepends: _*) enablePlugins JavaAppPackaging

// Extensions

lazy val aws = project in file("extensions/aws") settings(commonSettings: _*) dependsOn core

// Contrib

lazy val slick = project in file("contrib/slick") settings(commonSettings: _*) dependsOn core

bulid.sbt

name := "h-core"

libraryDependencies ++= Seq(
  "org.apache.commons"          %   "commons-compress"          %   "1.9",
  "com.jsuereth"                %%  "scala-arm"                 %   "1.4"
)

build.sbt

name := "h-client"

libraryDependencies ++= Seq(
  "jline"                       %   "jline"           %   "2.12",
  "com.typesafe.play"           %%  "play-ws"         %   "2.4.1"
)

build.sbt name := "h-master"

libraryDependencies ++= Seq(
  "com.typesafe.play"           %%    "play"                      %   "2.4.1",
  "com.typesafe.play"           %%    "play-netty-server"         %   "2.4.1",
  "com.leansoft"                %     "bigqueue"                  %   "0.7.0"
)

resolvers += "github.release.repo" at "https://raw.githubusercontent.com/bulldog2011/bulldog-repo/master/repo/releases/"

checksums in update := Nil

Update 2 tried the solution given my user rosshsr but error remain same 

libraryDependencies +=  excludeAll(

  ExclusionRule("com.esotericsoftware.minlog" , "minlog"),
  ExclusionRule("org.slf4j" ,   "slf4j-nop"),
  ExclusionRule("com.esotericsoftware.kryo" % "kryo")

)

Please help

scala sbt sbt-0.13
Original Q&A
1

There are 1 best solutions below

1
turbo_laser On

Since I'm not familiar with where in all of these libraries you're depending on these sets of clashes are occurring, I'll just give you my general strategy for fixing this.

You need to exclude one of those clashing dependencies.

libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.0.0-alpha" excludeAll(

  ExclusionRule("commons-logging","commons-logging-api"),
  ExclusionRule("asm","asm")

)

This is an excerpt from one of my build.sbt files.

You would need to attach that excludeAll bit to the dependency and add something like ExclusionRule("org.slf4j","slf4j-simple")

Related Questions in SCALA

Related Questions in SBT

Related Questions in SBT-0.13

Trending Questions

Popular # Hahtags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json

Popular Questions

.

Copyright © 2021 Jogjafile Inc.