How to fetch more than 50 items when executing nexus rest api get request

25 Views Asked by At

we have a corporate remote nexus repository.i want a groovy (which will be added to jenkins)script get the paths of all the directories and subdirectories and put into a list and return the list.Based on the selection of path with jenkins choise parameter i want to list all the zip files form the seclected path and the selected zip file will be used when the jenkins job runs.

My jenkins job selction

jenkins active choise parameter selection

so first to get all the directories and subdirectories path i wrote a groovy scrip which sends a rest api get requrest to my nexus repository

def requestUrl = "${nexusbaseUrl}/service/rest/v1/search?repository=<myreponame>&group=/<myGroupName>/*"
def response = "curl -X GET $requestUrl".execute().text

then i am parsing the response and adding value of the key called "group" to a from the response . the value will be path to directory.

now the issue is nexus get api returns only 50 items/components with continuation token at the end . I know from Nexus sonetype documentation we can add this token at the end of request url to get next 50 data but as my data is very large i cant afford to make many requests and this will take more time.

My remote repo structure

|-myGroupName
    |--project1
    |--project2
          |-sub1
          |-sub2
          |-sub3
             |-v1.0
             |-v2.0
                 |-2024-1-3
                 |-2024-1-5
                     |-bbc.zip
             |-v3.0
          |-sub4
      |--project3

My rest api response

    {
  "items" : [ {
    "id" : "some data",
    "repository" : "some data",
    "format" : "raw",
    "group" : "/myGroupName/project2/sub3/v2.0/2024-1-5/",
    "name" : "/myGroupName/project2/sub3/v2.0/2024-1-5/bbc.zip",
    "version" : "some data",
    "assets" : [ {
      "downloadUrl" : "",
      "path" : "/myGroupName/project2/sub3/v2.0/2024-1-5/bbc.zip",
      "id" : "some data",
      "repository" : "myreponame",
      "format" : "raw",
      "checksum" : {
        "sha1" : "some data",
        "md5" : "some data"
      },
      "contentType" : "application/zip",
      "lastModified" : "some data",
      "lastDownloaded" : "some data",
      "uploader" : "some data",
      "uploaderIp" : "some data",
      "fileSize" : 0
    } ],
    "tags" : [ ]
  }, {49 other items} ],
  "continuationToken" : "938580409939430"
}

some data here is confidential information. As I said I am using this continuation token to fetch next 50 data and so on...

My groovy code

@Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.7.1')
 
import groovy.json.JsonSlurper
 
def getFilesRecursive(String baseUrl) {
    def fileSet = new HashSet<>() // Using Set to ensure unique entries
    def continuationToken = null
 
    do {
        def response = makeGetRequest(baseUrl, continuationToken)
        def json = new JsonSlurper().parseText(response)
 
        json.items.each { item ->
            fileSet.add(item.group) // Adding path to the set
        }
 
        continuationToken = json.continuationToken
    } while (continuationToken)
 
    return fileSet as List // Converting Set to List before returning
}
 
def makeGetRequest(String url, String continuationToken = null) {

    def tokenParam = continuationToken ? "&continuationToken=${continuationToken}" : ""
    def requestUrl = "${url}${tokenParam}"
    println "requesturl is " + requestUrl
    def response = "curl -X GET $requestUrl".execute().text
    return response
}
 
def main() {
    long startTime = System.currentTimeMillis() // Record start time
    def nexusbaseUrl = 'https://<myBaseurl>'
    def requestUrl = "${nexusbaseUrl}/service/rest/v1/search?repository=<myreponame>&group=/<myGroupName>/*"
 
    def fileList = getFilesRecursive(requestUrl)
 
    println "List of files:"
    fileList.each { println it }
    long endTime = System.currentTimeMillis() // Record end time
    long totalTime = endTime - startTime
    println "total time in sec: " + totalTime/1000
    
}
 
main()

is there a way to get more than 50 items (say 1000) items with one get request or is there a better way to get path to all the directories of remote nexus server recursively and add them to a list. (groovy script is appreciated) ??

0

There are 0 best solutions below