Git add subtree of repo with removed LFS files

Question

I have a few repos that I want to consolidate into a single one but maintain all git history. I am not interested in using submodules or repo tools, I just want to make a monolithic repo for this. For the example let's say the repos are named like this and I want to put them in the same structure in the new repo:

├ includes
├ source
├ third-party <- contains git-lfs files
└ asset  <- contains git-lfs files

For the repos that did not have any git lfs files in them it was easy, I just ran:

$ git subtree add -P includes [email protected]:org/includes.git  main
$ git subtree add -P source [email protected]:org/source.git  main

But when I do that with a repo that has git-lfs files in it I get this error:

$ git subtree add -P third-party [email protected]:org/third-party.git  main
git fetch [email protected]/third-party.git  main
From gitlab.com:org/third-party 
 * branch            main       -> FETCH_HEAD
Downloading third-party/prometheus-cpp-1.0.1.tar.gz (13 MB)
Error downloading object: third-party/prometheus-cpp-1.0.1.tar.gz (053f865): Smudge error: Error downloading third-party/prometheus-cpp-1.0.1.tar.gz (053f865548c7a199107f88c1f0f7a399fb75f933f9a7759e1eabcf37c261c3e6): [053f865548c7a199107f88c1f0f7a399fb75f933f9a7759e1eabcf37c261c3e6] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it

Errors logged to '.git/lfs/logs/20230708T125153.937604803.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: third-party/prometheus-cpp-1.0.1.tar.gz: smudge filter lfs failed

So that led me to discover that subtrees with git-lfs is not support: git lfs not working with git subtree ? #854. I've tried a number of ways to work around this, but my latest attempt is the following. Since I don't really care about the history of the git lfs files, since they are all binaries, and there are only a handful of them, I am trying to create a local clone of the third-party repo and use bfg to remove them completely (per How to remove file from Git history?), which seems to succeed. I confirmed with git log --all --full-history -- googletest-1.13.0.tar.gz they aren't found. Then I try to pull it into the new monolithic repo from my local repo with $ git subtree add -P third-party ../third-party_wo_lfs main but it is still fails and seems like it is still trying to pull from gitlab:

$ git subtree add -P third-party ../third-party_wo_lfs  main
git fetch ../third-party_wo_lfs main
From ../third-party_wo_lfs
 * branch            main       -> FETCH_HEAD
Downloading third-party/googletest-1.13.0.tar.gz (863 KB)
Error downloading object: third-party/googletest-1.13.0.tar.gz (ad7fsba): Smudge error: Error downloading third-party/googletest-1.13.0.tar.gz (ad7fdba11ea011c1d925b32d9cf4af2c66a352e18d4c7264392fead75e919363): [ad7fdba11ea011c1d925b32d9cf4af2c66a352e18d4c7264392fead75e919363] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it

Errors logged to '.git/lfs/logs/20230708T131124.76575032.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: third-party/googletest-1.13.0.tar.gz: smudge filter lfs failed

I even ran git remote remove origin in the third-party_wo_lfs repo but it I get the same error, and confirmed by killing my VPN, which I need to access gitlab, and the command hangs. So how can I do what I am trying to do? I am open to other recommendations as well, I just want to maintain the git history. Thank you.

Answer 1

For anyone who comes across this question in the future, I was able to use this command to successfully import the subtrees

git filter-repo --force --blob-callback "
if blob.data.split(b'\n', 1)[0] == b'version https://git-lfs.github.com/spec/v1':
  blob.skip()
"

Explanation:

• git filter-repo rewrites git history
• --force is needed to skip safety checks
• if blob.data.split(b'\n', 1)[0] == b'version https://git-lfs.github.com/spec/v1': blob.skip() This script checks if the first line of the file contains an LFS signature and skips adding this file in such a case.

Source: https://stackoverflow.com/a/73743529/1556142