Migrating projects from StarTeam to Bitbucket

Question

We have around 5000 projects (various technologies) managed in StartTeam. Client moving to a new stack Bitbucket, JFrog, Bamboo & UrbanCode.

Bitbucket: It will be used for source control and underlying GIT will be used as revision control systems.

JFrog: Will manage the binaries for maven repository.

Bamboo: Will be used for build server.

UrbanCode: It will be used to automate the code deployment process.

I am taking JAVA project as reference for my questions. Currently StartTeam project contains source code as well as all required binaries and its not a maven project. ANT script is used for project build.

Requirement is migrating project like this to Bitbucket with minimal effort. the Bitbucket should not contain any binaries it would only manage the source code. Client also has setup an artifactory JFrog which will manage binaries for maven.

As part of this migration I am thinking a hybrid approach something like:

Step 1: Project will be downloaded from StartTeam.

Step 2: All binaries will be added to a new pom.xml as dependencies

Step 3: Code will be checkin to Bibucket

Step 4: In bamboo build server the build will be configured in two steps

a. First it will download all the required jars into a folder by executing the pom.xml

b. Then the existing ant script will be called to build the project by adding all the jars downloaded in previous step into CLASSPATH

Step5: UrbanCode will be configured to automate the deployment process

Already migrated few projects using this approach.

If time permits may be we will consider first fully mavenized the project (secondary approach) before importing into Bitbucket.

Questions:

1) There are approximately 5000 projects that need to be migrated so I am looking for expert suggestions how to proceed with approach 1 (hybrid approach)?

2) Please suggest is there any other approaches which can make this migration with less effort?

3) Any tools which can accelerate this migration?

Answer 1

Finally I have automated this migrations process as much as possible in the following way:

1) Download the project from StarTeam

2) I have Developed a JAVA utility which will scan the project workspace and dump all the jar details into an excel sheet. For each jar it will calculate the checksum (SHA-1) using the below code

public static String calculateChecksum(File javaJarFile) {

    String filepath = javaJarFile.getAbsolutePath();
    StringBuilder sb = new StringBuilder();
    FileInputStream fis = null;

    try {
        MessageDigest md = MessageDigest.getInstance("SHA-1"); //40 character checksum
        fis = new FileInputStream(filepath);
        byte[] dataBytes = new byte[1024];
        int nread = 0;

        while ((nread = fis.read(dataBytes)) != -1)
            md.update(dataBytes, 0, nread);

        byte[] mdbytes = md.digest();

        for (int i = 0; i < mdbytes.length; i++)
            sb.append(Integer.toString((mdbytes[i] & 0xff) + 0x100, 16)
                    .substring(1));

    } catch (NoSuchAlgorithmException e) {
        e.printStackTrace();

    } catch (IOException e) {
        e.printStackTrace();

    } finally {
        try {
            if (fis != null)
                fis.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    return sb.toString();
}

Then it will query the Artifactory using the checksum value for the matching jar details like groupId, artifactId, version etc...If the jar details not found in Artifactory, then it will query MAVEN central repository for the matching jar details. Finally all the existing project jar details and the corresponding MAVEN compatible jar details will be dumped in the excel sheet.

Sample code for querying the MAVEN central repository

        CloseableHttpClient httpClient = HttpClients.custom().build();
        HttpPost getRequest = new HttpPost("http://search.maven.org/solrsearch/select?q=1:<JAR CHECKSUM>&rows=1&wt=json");
        getRequest.addHeader("accept", "application/json");

        HttpResponse response = httpClient.execute(getRequest);

        if (response.getStatusLine().getStatusCode() != 200) {
            throw new RuntimeException("Failed : HTTP error code : "
                    + response.getStatusLine().getStatusCode());
        }

        BufferedReader br = new BufferedReader(new InputStreamReader(
                (response.getEntity().getContent())));

        String output;
        StringBuffer outputBuffer = new StringBuffer("");
        while ((output = br.readLine()) != null) {
            outputBuffer.append(output);
        }

        JSONObject jsonObj = new JSONObject(outputBuffer.toString());
        LOGGER.info("MAVEN Compatible Dependency Found: " + jsonObj.getJSONObject ("response").getInt("numFound"));

        if (jsonObj.getJSONObject ("response").getInt("numFound") > 1) {
            JSONArray jsonArray = jsonObj.getJSONObject ("response").getJSONArray("docs");
            JSONObject object = (JSONObject) jsonArray.get(0);
            LOGGER.info(object.getString("g"));
            LOGGER.info(object.getString("a"));
            LOGGER.info(object.getString("v"));
        }

Sample code for querying the Artifactory

String checkSumUri = "https://anupg.jfrog.io/anupg/api/search/checksum?sha1=" + checkSum;
HttpResponse rtFactResponse = callService (checkSumUri, true);
BufferedReader br = new BufferedReader(new InputStreamReader((rtFactResponse.getEntity().getContent())));

String output;
StringBuffer outputBuffer = new StringBuffer("");
while ((output = br.readLine()) != null) {
    outputBuffer.append(output);
}

String uri = null;
if (!outputBuffer.toString().trim().equals("")) {
    JSONObject jsonObj = new JSONObject(outputBuffer.toString());
    JSONArray jsonArray = jsonObj.getJSONArray("results");

    for (int i=0; i < jsonArray.length(); i++) {
        JSONObject jsonUriObject = (JSONObject) jsonArray.get(i);
        System.out.println("URI---------->" + jsonUriObject.getString("uri").replace(".jar", ".pom"));
        uri = jsonUriObject.getString("uri").replace(".jar", ".pom");
    }
}

if (uri != null) {

    String downloadURI = null;
    HttpResponse uriResponse = callService (uri);

    BufferedReader uriBR = new BufferedReader(new InputStreamReader(
            (uriResponse.getEntity().getContent())));

    String uriOutput;
    StringBuffer uriOutputBuffer = new StringBuffer("");
    while ((uriOutput = uriBR.readLine()) != null) {
        uriOutputBuffer.append(uriOutput);
    }

    if (!uriOutputBuffer.toString().trim().equals("")) {
        JSONObject jsonUriObject = new JSONObject(uriOutputBuffer.toString());
        System.out.println("Download URI---------->" + jsonUriObject.getString("downloadUri"));
        downloadURI = jsonUriObject.getString("downloadUri");
    }

    HttpResponse downloadUriResponse = callService (downloadURI, true);

    if (downloadUriResponse.getStatusLine().getStatusCode() != 200) {
        throw new RuntimeException("Failed : HTTP error code : "
                + downloadUriResponse.getStatusLine().getStatusCode());
    }   

    InputStream is = downloadUriResponse.getEntity().getContent();
    DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
    Document doc = null;

    try {
        DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
        doc = dBuilder.parse(is);
        doc.getDocumentElement().normalize();
    } catch (ParserConfigurationException e) {
        e.printStackTrace();
    } catch (SAXException e) {
        e.printStackTrace();
    } finally {
        if (is != null) {
            is.close();
        }
    }

    System.out.println("root of xml file: " + doc.getDocumentElement().getNodeName());
    Element element = doc.getDocumentElement();

    if (getNodeValue("groupId", element) != null && 
            getNodeValue("artifactId", element) != null && 
            getNodeValue("version", element) != null) {
        metadata.setGroupId(getNodeValue("groupId", element));
        metadata.setArtifactId(getNodeValue("artifactId", element));
        metadata.setVersion(getNodeValue("version", element));
        metadata.setRTfactoryFound(true);
    }
}

3. There could be a chance that project may contain custom jars which may not be available either in Artifactory or MAVEN central. In such cases the generated excel sheet will be shared with AD team for corresponding jar details. The jar will be installed in Artifactory and the AD team will update the excel sheet with the groupId, arfactId, version.

4. Once the excel sheet having all the jar details the JAVA utility will generate the pom.xml by reading the corresponding MAVEN details. The "copy-rename-maven-plugin" plugin is used here to copy all the maven downloaded jars from "target/dependency" folder to respective project folder. This pom will be configured as the first build steps in Bamboo server followed by build.xml which will build the project. Note I am using hybrid approach in build process.

Below is the code snippets for the same

    Writer w = null;
    MavenXpp3Writer mavenXpp3Writer = new MavenXpp3Writer();

    try {
        List<JarDetails> uniquejarDetails = removeDuplicateJars (jarDetails);

        w = WriterFactory.newWriter (new File(location + "pom.xml"), "UTF-8");

        Model model = new Model();
        model.setGroupId("com.projectname");
        model.setArtifactId("project-analyzer");
        model.setVersion("1.0");
        model.setModelVersion("4.0.0");

        List<Dependency> dependencies = new ArrayList<Dependency>();            

        Plugin copyDependency = new Plugin();
        copyDependency.setGroupId("org.apache.maven.plugins");
        copyDependency.setArtifactId("maven-dependency-plugin");
        copyDependency.setVersion("2.10");

        PluginExecution copyDependencyPluginExecution = new PluginExecution();
        copyDependencyPluginExecution.setId("copy-dependencies");
        copyDependencyPluginExecution.setPhase("generate-sources");

        List<String> copyDependencyGoalsList = new ArrayList<String>();
        copyDependencyGoalsList.add("copy-dependencies");
        copyDependencyPluginExecution.setGoals(copyDependencyGoalsList);


        Plugin plugin = new Plugin();
        plugin.setGroupId("com.coderplus.maven.plugins");
        plugin.setArtifactId("copy-rename-maven-plugin");
        plugin.setVersion("1.0.1");

        PluginExecution pluginExecution = new PluginExecution();
        pluginExecution.setId("copy-jars");
        pluginExecution.setPhase("generate-sources");

        List<String> goalsList = new ArrayList<String>();
        goalsList.add("copy");
        pluginExecution.setGoals(goalsList);

        String domString = "<configuration><fileSets>";

        for (int jarCount = 0; jarCount < uniquejarDetails.size(); jarCount++) {
            if (uniquejarDetails.get(jarCount).getDependencyMetadata().getGroupId() != null) {

            Dependency dependency = new Dependency();

            dependency.setGroupId(uniquejarDetails.get(jarCount).getDependencyMetadata().getGroupId());
            dependency.setArtifactId(uniquejarDetails.get(jarCount).getDependencyMetadata().getArtifactId());
            dependency.setVersion(uniquejarDetails.get(jarCount).getDependencyMetadata().getVersion());
            dependencies.add(dependency);

            //Add copy-rename-maven-plugin configurations
            String mavenJarName = uniquejarDetails.get(jarCount).getDependencyMetadata().getArtifactId() + "-"
                    + uniquejarDetails.get(jarCount).getDependencyMetadata().getVersion() + ".jar";
            String mavenJar = "target/dependency/" + mavenJarName;

            domString += "<fileSet><sourceFile>" + mavenJar + "</sourceFile>";
            domString += "<destinationFile>" + new File(uniquejarDetails.get(jarCount).getJarRelativePath() + 
                    "/" + mavenJarName) + "</destinationFile></fileSet>";
            }
        }

        domString += "</fileSets></configuration>";

        InputSource is = new InputSource();
        is.setCharacterStream(new StringReader(domString));
        Xpp3Dom dom = Xpp3DomBuilder.build(new StringReader(domString));

        pluginExecution.setConfiguration(dom);

        List<PluginExecution> pluginExecuionList = new ArrayList<PluginExecution>();
        pluginExecuionList.add(pluginExecution);

        List<PluginExecution> copyDependencyPluginExecuionList = new ArrayList<PluginExecution>();
        copyDependencyPluginExecuionList.add(copyDependencyPluginExecution);

        plugin.setExecutions (pluginExecuionList);
        copyDependency.setExecutions (copyDependencyPluginExecuionList);

        List<Plugin> pluginList = new ArrayList<Plugin> ();
        pluginList.add(copyDependency);
        pluginList.add(plugin);

        Build build = new Build();
        build.setPlugins(pluginList);

        model.setDependencies(dependencies);
        model.setBuild(build);

        mavenXpp3Writer.write(w, model);

    } catch (UnsupportedEncodingException e) {
        LOGGER.error(e.getMessage(), e);

    } catch (FileNotFoundException e) {
        LOGGER.error(e.getMessage(), e);

    } catch (IOException e) {
        LOGGER.error(e.getMessage(), e);

    } catch (FactoryConfigurationError e) {
        e.printStackTrace();

    } catch (XmlPullParserException e) {
        e.printStackTrace();

    } finally {
        try {
            if (w != null)
                w.close();
        } catch (IOException e) {
            LOGGER.error (e.getMessage(), e);
        }
    }

5. Next steps is cleanup: It will delete all the jars from the project and will keep a track of the jar locations.

6. Created a shell script which will connect the bitbucket server and will push the code using git command. I am using "GIT Bash" to execute the script. Note, I am using Bitbucket rest API to create the project and repositories remotely before pushing the code.

REST service details for creating the project:

curl -X POST -v -u $bitbucket_user:${bitbucket_password} -H "Content-Type: application/json" "http://hostname:7990/rest/api/1.0/projects" -d "{\"key\": \"$project_key\",\"name\": \"$project_name\", \"description\": \"$project_desc\"}" > response.json

REST service details for creating repository under the above Project

curl -X POST -v -u $bitbucket_user:${bitbucket_password} -H "Content-Type: application/json" "http://hostname:7990/rest/api/1.0/projects/$project_key/repos" -d "{\"name\": \"$repository_name\", \"scmId\": \"git\", \"forkable\":true}" > repo-response.json

Git commands for pushing the project into Bitbucket

git init
git add .
git commit -m "Initial commit"
git remote add origin  http://$bitbucket_user:${bitbucket_password}@hostname:7990/scm/${project_key}/${repository_name}.git
git push -u origin --all

7. Next step is bamboo server and UrbanCode configurations which I am doing manually as bamboo server does not exposed any configurations REST API. There are REST services only for read only operations.

Using this approach we have automated 80% of the migration activity.