How can I copy code from one Code Repository to another in Foundry?

Question

I have a lot of code in one Code Repository that I would like to move to a different repository, but I don't really want to copy paste file by file. I know that the repos are git based, but what exactly would I need to copy?

For example

# Copy code from RepoA
/transforms-python/src/myproject/**/*.py

# to RepoB
/transforms-python/src/myproject/**/*.py

Answer 1

Before diving into the details, it’s worth reflecting on why it may be necessary to copy code from one Code Repository to another. For example:

• If you need to move a Code Repository then press File > Move in the menu bar. If you move the Code Repository out of the current Project you will need to update your Foundry Project references.
• If the purpose is to leverage user-written functions in another Code Repository (e.g. a set of cleaning functions), then you should consider writing a Library. Libraries can be created by selecting the Libraries template in the Code Repositories splash screen, and can be used in all Code Repositories without needing to copy code multiple times.

Let’s now assume that the reason for copying code is a good one, and you’re trying to copy code from RepoA to RepoB.

Here’s the whole terminal flow upfront, below you’ll find steps that explain it:

# clone (step 1 to 3)
$ git clone https://repoA
$ git clone https://repoB

# copy files (step 4) 
# stolen from https://unix.stackexchange.com/questions/83593/copy-specific-file-type-keeping-the-folder-structure )
$ find . -name '*.py' -exec cp --parents \{\} /target \;

# edit your outputs (step 5) 
# this line opens a folder in sublime text
$ subl -a repoB/

# after saving commit and push (step 6)
git commit -a -m "copied code"
git push origin master

Step-by-step explanation:

1. In Foundry, create a new Code Repository called RepoB
2. Clone (https://git-scm.com/docs/git-clone) RepoA to your local machine using git clone in a command line interface (CLI) (https://www.codecademy.com/article/command-line-interface)
   1. You will need to get the remote URL by pressing the Clone button in the menu bar in Code Repositories
   2. Alternatively, if the number of files you need to copy is small, consider copy-pasting each file within the Code Repositories UI
3. Clone RepoB to your local machine using git clone in a command line interface (CLI) (https://www.codecademy.com/article/command-line-interface)
4. Copy only the files that produce datasets from the RepoA folder to RepoB folder on your local machine
   1. You should not copy any hidden or non-src files from repoA to repoB (this includes, but is not limited to: meta.yaml, build.gradle, templateConfig.json, gradlew, and so on).
      1. If you are unsure which files count as hidden, go to your newly created repoB, expand the file tree so you can see all files and folders in the repo, then toggle on “show hidden files” using the gear icon next to the branch selector. Hidden files will show up when you turn on that toggle and have a lighter filename color than non-regular files.
5. Update the dataset output paths in RepoB on your local machine
   1. If the output paths of the datasets don't change, then you'll run into a problem where CI fails in RepoB due to the dataset being "owned" by RepoA. This will require complex manual remediation.
6. Add (https://git-scm.com/docs/git-add), commit (https://git-scm.com/docs/git-commit), and push (https://git-scm.com/docs/git-push) all changes to RepoB to the remote branch (https://git-scm.com/book/en/v2/Git-Branching-Remote-Branches)

Additional things to consider:

• If your code is dependent on packages, then you should go to the Libraries section in the Code Repositories left-hand sidebar and manually add the relevant packages
• If RepoB is in a different Project to RepoA, then you will also need to reference all input datasets in the new Project by pressing the lightbulb icon next to the dataset Input statement (or by adding the reference in the Project details page in Compass)
• If your code utilizes custom Spark profiles, then you will also need to configure those in the Code Repository tab Settings > Spark