Gradle - Download all version jars of a single groupID:artifactID from Artifactory

Question

I use Gradle to build a project (Java).

Gradle 2.3, Java7/8. Works great.

To build a project, we define what all libraries we need during compile, test (testRuntime) and runtime stages and we define all those libraries in dependencies section like:

dependencies {
    compile 'some_groupID:some_artifactID:some_version_x.y.z@ext_if_any'

    testRuntime 'SomeOtherOrSame_groupID:some_other_or_same_artifactID:some_version_x.y.z@ext_if_any'

    runtime 'SomeOtherOrSame_groupID:some_other_or_same_artifactID:some_version_x.y.z@ext_if_any'

}

We have multiple such entries in the dependencies section for all the libraries that our app/service would need for running build/test/IT tests etc (at runtime).

Gradle works good so far and downloads all the artifacts from Artifactory.

I'm working on a project where I need to get all the versions of an artifact (what all is available in the dependencies section) for a single groupID:artifactID only) i.e. I want the following .jar (default extension) and create a zip file containing all the versioned .jars in it:

compile 'my.company.com:cool_library:1.0.0'
compile 'my.company.com:cool_library:1.0.1'
compile 'my.company.com:cool_library:1.1.0'
compile 'my.company.com:cool_library:1.2.0'

runtime 'my.company.com:another_cool_library:1.0.0'
runtime 'my.company.com:another_cool_library:1.2.0'
runtime 'my.company.com:cool_library:1.0.1'

I know how to create a .zip file in Gradle (it's easy) but Gradle is NOT getting/downloading all the .jar (for every version as I have listed above) from Artifactory or from a given binary repository system.

It only gets the latest one (i.e. cool_service-1.2.0.jar). Gradle does this nice because, this way you won't have duplicate class in the class path coming from the same project (cool_library) and due it its different versions mentioned in the dependencies section.

One would say, why I would need OLDER versions when the latest/greatest code is there in version 1.2.0 of cool_library and if a project which is consuming this library wants it, just say I want my.company.com:cool_library:1.2.0 and be done with it or get whatever single version you want for compiling. In my case, DEV team have written application code in a way that they define cool_library-1.0.0.jar here, but cool_library-1.2.0.jar somewhere else. Basically, I need all what a Developer has defined in the build.gradle file.

How can I create a custom_zip.zip file which will have all the cool_library-x.y.z.jar files that I have mentioned in dependencies section for "compile" heading.

Thanks.

Answer 1

Using this great tool, it's possible. Read it all (for different ways to do things).

Link Gradle Download Task: https://github.com/michel-kraemer/gradle-download-task

For ex: In your build.gradle you'll have this:

plugins {
    id "de.undercouch.download" version "1.2"
}

import de.undercouch.gradle.tasks.download.Download
apply plugin: 'java'
apply plugin: 'de.undercouch.download'


repositories {
  maven {
    url "http://yourartifatory:1020/artifactory/virtual-repos"
  }
}

//Lets says, I want to download my.company.com:CoolServices:1.0 jar or any extension file
//Then I can create a list and specify what all I need as:
def deps = [
            [ "my/company/com", "CoolServices", "1.0", "jar" ],
            [ "my/company/com", "CoolServices", "1.1", "jar" ],
            [ "my/company/com", "CoolServices", "1.2", "jar" ]
           ]

task downloadArts << {
     //Iterate over each dependencies as per the list
     deps.each { groupId, artifactId, version, extn ->
          download {
              src "http://yourartifactory.fqdn.com:1020/artifactory/simple/some-physical-actual-repository-local/$groupId/$artifactId/$version/$artifactId-$version.$extn"
              dest buildDir
              //If you want the downloaded files inside build/xxx folder, create that xxx folder first outside of the clousure i.e. file("build/xxx").mkdir()
          }
     }
}

Now, run "gradle clean downloadArts" and it'll print what it's going to download / line and inside either build or build/xxx folder where you'll find all of your downloaded files (.jar/etc extensioned files that you mentioned in the deps list variable).

You can also use it to push into dependencies section for compile, runtime, testRuntime OR create a custom_zip.zip file if required.

For ex:

dependencies {
    compile  fileTree( dir: "build", include: '*.*' )
    runtime  fileTree( dir: "build/xxx", include: '*.*' )
}

Or create a custom_zip.zip file (containing all the downloaded .jar/extension files in it).

Answer 2

I have a gradle project that I use for downloading specific dependency releases into a directory, which I would then upload to our local IVY repo so that we control and have available particular versions of software in case they disappear.

The gist of it is using ant.retrieve to pull the files down from the external repo for the version you're after into a directory, and then do with them as you will.

This isn't the complete script, but enough of it to work I hope. Ant will use the ivy-cache dir for a cache, and your 'repo' containing all your jar files will be in the ivy-repo dir. You can adjust the antRetrieve task to flatten your file structure (change the pattern variables), and remove the ivy/src files as you wish, but I present you the script as I have it:

project.ext.REPO_DOWNLOADER_DIR = "ivy-repo"
project.ext.IVY_SETTINGS = 'ivy-settings.xml' 
project.ext.IVY_CACHE = file('ivy-cache')

dependencies {
    compile ':ivy:2.3+'
}

def antRetrieve(dep) {
    // in each of the following patterns, the file will be downloaded with the [name] values filled in correctly,
    // e.g. from ant:retrieve(..."src,source"), the [type] will be either "src" or "source" depending on which was available to download.
    def jarPattern = "$REPO_DOWNLOADER_DIR/[organisation]/[revision]/[module]/[type]s/[artifact]-[revision].[ext]"
    def srcPattern = "$REPO_DOWNLOADER_DIR/[organisation]/[revision]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"
    def docPattern = "$REPO_DOWNLOADER_DIR/[organisation]/[revision]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"
    def ivyPattern = "$REPO_DOWNLOADER_DIR/[organisation]/[revision]/[module]/ivy.xml"
    def (org, module, rev) = dep.split(':')

    println "retrieving $org:$module:$rev"

    ant.retrieve(inline: "true", type: "jar,bundle", transitive: "true", pattern: "$jarPattern", ivypattern: "$ivyPattern", organisation: "$org", module: "$module", revision: "$rev", conf: "runtime,default")
    ant.retrieve(inline: "true", type: "src,source,sources", transitive: "true", pattern: "$srcPattern", organisation: "$org", module: "$module", revision: "$rev", conf: "sources")
    ant.retrieve(inline: "true", type: "doc,docs,javadoc,javadocs", transitive: "true", pattern: "$docPattern", organisation: "$org", module: "$module", revision: "$rev", conf: "javadoc")

}

task retrieve << {
    ant.taskdef(resource: 'org/apache/ivy/ant/antlib.xml', classpath: configurations.compile.asPath)
    ant.properties['repo.downloader.cache'] = IVY_CACHE
    ant.settings(file: IVY_SETTINGS)

    if (project.hasProperty('modules')) {
        def moduleList = project.modules.split(',')
        moduleList.each { module ->
            antRetrieve(module)
        }
    }

    if (project.hasProperty("modfile")) {
        def modulesFile = file(project.modfile)
        if (! modulesFile.exists()) {
            throw new GradleException("Could not find specified modules file: $modulesFile")
        }
        modulesFile.eachLine { module ->
            antRetrieve(module)
        }
    }
}

Then the ivy-settings.xml file sets up the external repos you want ant to pull from. Add any additional as you see fit (e.g. clojars for clojure jars):

<ivysettings>
    <settings defaultResolver="spring-chain" />
    <caches defaultCacheDir="${repo.downloader.cache}" />
    <resolvers>
        <chain name="spring-chain">
            <url name="com.springsource.repository.bundles.release">
                <ivy pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
                <artifact pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
            </url>
            <url name="com.springsource.repository.bundles.external">
                <ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
                <artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
            </url>
            <ibiblio name="ibiblio" m2compatible="true" />
            <ibiblio name="uk-maven" m2compatible="true" root="http://uk.maven.org/maven2"/>
        </chain>
    </resolvers>
</ivysettings>

You can call the whole thing with one of following:

./gradlew retrieve -Pmodules='a:b:1.0,x:y:1.1'
./gradlew retrieve -PmodFile='/path/to/modlist.txt'

which allows you to dump all the versions into a text file, or specify them on the command line.

This will pull dependencies down too. The only thing it doesn't do is pull the src files for dependencies (it does for the main named archive), but I have a shell script that just interrogates dirs looking for missing src and iterates the retrieve task with those as main entries, but if you're not after sources, then you should be fine.

There's still some work to do if you adopt this strategy, but it will download all the files for any dependency you want, and you can go off and zip them all up as required for your own consumption. The above just works for uploading into a local ivy repo directly.