git branching advanced usage, rebase from upstream branch

Question

Following up on Git: rebase onto development branch from upstream, basically the same ask as:

I have local master and develop branches. I do all my work on develop and then merge them into master for releases. There is a remote branch, upstream/master which has changes I want, but I want to rebase my changes in develop (which shares a common ancestor) on top of its changes and put them back into develop. I've already done git fetch upstream.

Here is the (a bit more complicated/advanced) situation I'm facing with, starting from scratch.

1. I forked an upstream, and made my own changes from its development branches (dev/br-1), in my own new branch (dev/br-2), and git push to my own repo.
2. Now, upstream has advanced, both in its master and develop branches.
3. The way upstream advanced its develop branch is via rebase, i.e., all its own changes are put back on top after rebase.
4. My local repo is gone with my old machine, and I need to pick-up/git-clone from my own repo to continue with my customization.
5. I want to rebase my changes in my dev/br-2 branch (which shares a common ancestor) on top of all the upstream changes.
6. I've already done git fetch upstream and was able to rebase my master with upstream's.
7. It is how to rebase my current dev/br-1 from upstream, then dev/br-2 from dev/br-1 branch that makes my head spinning nonstop.

Although it looks like a very specific case, but the principle of how to do git branching and rebasing still applies, and the answer would be very educational to the general public. Please help.

UPDATE: So I looked at @torek suggested git rebase --onto command, like How to git rebase a branch with the onto command?, and all their refed docs, and think my problem is still one or two levels up than what I've read (because there are two repos and 5 branches involved). Here is the summary:

Situation at point#1:

A---B---C---D  master (upstream)
    \
     E---F---G  dev/br-1 (upstream)
             \
              H---I---J dev/br-2 (myown)

Situation at point#2&3:

A---B---C---D  master (upstream)
            \
              E'---F'---G'  dev/br-1 (upstream)

And I don't even know where should I draw my myown branch. Here is my current situation (which may have already been messed up, as I see HEAD might be in a weird place):

$ git log --all --decorate --oneline --graph
* 7aec18c (upstream/dev/br-1) more updates
* b83c3f8 more updates
* 4cf241f update file-a
* 200959c update file-a from main
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (HEAD -> dev/br-1, origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

The ultimate goal is to place myown

H---I---J dev/br-2

branch on top of the newly rebased G', in my own repo, after catching up with upstream. I.e., in my own repo, in the end, it should look like this:

A---B---C---D  rebased master (upstream)
            \
             E'---F'---G'  rebased dev/br-1 (upstream)
                       \
                        H'---I'---J' rebased dev/br-2

How to do that?

More explain by command:

 cd /tmp
 mkdir upstream
 cd upstream
 # prepare its `master` and `dev/br-1` branches

 cd /tmp
 git clone upstream myfork
 # prepare my own changes based on the `dev/br-1` branch into `dev/br-2`

 cd /tmp/upstream
 # advance its `master` 
 . . .
 # and its`dev/br-1` branches
 git checkout dev/br-1
 git rebase -X theirs master dev/br-1
 . . .

Now upstream has advanced, both in its master and its develop branches (via rebase), and I need to pick-up from my own repo to continue with my customization.

 cd /tmp
 mv myfork myfork0
 git clone myfork0 myfork1
 cd myfork1
 git remote -v
 git remote add upstream /tmp/upstream
 git remote -v
 git fetch upstream
 git rebase upstream/master

 git checkout --track origin/dev/br-1

$ git remote -v
origin  /tmp/myfork0 (fetch)
origin  /tmp/myfork0 (push)
upstream        /tmp/upstream (fetch)
upstream        /tmp/upstream (push)

$ git branch -avv
* dev/br-1                  0006c5e [origin/dev/br-1] more updates
  master                    dc45a94 [origin/master: ahead 1] update file-a from main
  remotes/origin/HEAD       -> origin/master
  remotes/origin/dev/br-1   0006c5e more updates
  remotes/origin/dev/br-2   ce2a804 update file-a
  remotes/origin/master     2f5eaaf add file-a
  remotes/upstream/dev/br-1 7aec18c more updates
  remotes/upstream/master   dc45a94 update file-a from main

$ git log --all --decorate --oneline --graph
* 7aec18c (upstream/dev/br-1) more updates
* b83c3f8 more updates
* 4cf241f update file-a
* 200959c update file-a from main
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (HEAD -> dev/br-1, origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

UPDATE2:

With above status, when I tried --rebase-merges as suggested by @VonC, I'm getting:

$ git rebase --rebase-merges --onto master $(git merge-base dev/br-2 master) dev/br2
fatal: Not a valid object name dev/br-2
fatal: invalid upstream 'dev/br2'

$ git checkout --track origin/dev/br-2
Branch 'dev/br-2' set up to track remote branch 'dev/br-2' from 'origin' by rebasing.
Switched to a new branch 'dev/br-2'

$ git rebase --rebase-merges --onto master $(git merge-base dev/br-2 master) dev/br-2
Successfully rebased and updated refs/heads/dev/br-2.

$ git log --all --decorate --oneline --graph
* 344418c (HEAD -> dev/br-2) update file-a
* 4de3dec more updates
* 81af2ac more updates
* 1e3f9fb update file-a
| * 7aec18c (upstream/dev/br-1) more updates
| * b83c3f8 more updates
| * 4cf241f update file-a
| * 200959c update file-a from main
|/  
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (origin/dev/br-1, dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

Here, how to rebase my current dev/br-1 from upstream, then dev/br-2 from dev/br-1 branch please (detailed preparation can be found at https://pastebin.com/Df8VbCp2, if necessary).

Answer 1

Here is my current situation (which may have already been messed up, as I see HEAD might be in a weird place):

$ git log --all --decorate --oneline --graph
* 7aec18c (upstream/dev/br-1) more updates
* b83c3f8 more updates
* 4cf241f update file-a
* 200959c update file-a from main
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (HEAD -> dev/br-1, origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

OK, I might draw this horizontally as:

A   <-- origin/master
|\
| B--C--D   <-- dev/br-1 (HEAD), origin/dev/br-1
 \       \
  \       E   <-- origin/dev/br-2
   \
    F   <-- master, upstream/master
     \
      G--B'-C'-H--I   <-- upstream/dev/br1

A = 2f5eaaf add file-a; B = 85afa56 update file-a; and so on. I picked the later ones to be the "primes" based on subject lines, although that might be backwards: perhaps the one labeled B should be B' for instance. It seems likely that despite the same subject line for commits B and E, they have different patch-IDs (do different things to file-a), and it's hard to say whether matching up the two "more updates" commits here (calling the second one on the bottom row C') was right.

(Note that there's no dev/br2 here.)

But let's go back to this:

[We start with]

A--B--C--D   <-- master (upstream)
    \
     E--F--G   <-- dev/br-1 (upstream)
            \
             H--I--J   <-- dev/br-2 (myown)

I reformatted this slightly into my own preferred style (two dashes between commits, or one if using a prime marker to indicate that it's a copy).

The ultimate goal is to place myown

H--I--J   <-- dev/br-2 

branch on top of the newly rebased G', in my own repo, after catching up with upstream. I.e., in my own repo, in the end, it should look like this:

A--B--C--D   <-- rebased master (upstream)
          \
           E'-F'-G'   <-- rebased dev/br-1 (upstream)
                  \
                   H'-I'-J'   <-- rebased dev/br-2

The commits that must be copied are precisely E-F-G-H-I-J (in that order).

If you have all six commits in your own repository (plus of course the four A-B-C-D commits), then—ignoring the desired labels for a moment—all we need to do is convince Git to copy, as by cherry-picking, the six commits in question.

The two options for doing this are:

• git cherry-pick, which leaves both the originals and the copies for us to find easily; or
• git rebase, which does the copying, then moves one (1) branch name around for us.

Moving one branch name is insufficient. It's not particularly harmful and we could allow Git to do it, but let's just do this directly with cherry-pick. We'll start by checking out the commit onto which everything should land, creating a new branch name:

git switch -c copied <hash-of-D>

or:

git switch -c copied --no-track upstream/master

(assuming upstream/master names commit D). Then:

git cherry-pick <hash-of-D>..<hash-of-J>

or:

git cherry-pick upstream/master..dev/br2

(again the two names used here are just ways of typing in the raw commit hash IDs without having to type in raw commit hash IDs). The two-dot syntax here means any commits reachable from the second specifier, excluding all commits reachable from the first specifier, so this means commits from J all the way back to the root, minus commits from D all the way back to the root which means E-F-G-H-I-J.

The result of this copying would be:

           E'-F'-G'-H'-I'-J'  <-- copied (HEAD)
          /
A--B--C--D   <-- master (upstream)
    \
     E--F--G   <-- dev/br-1 (upstream)
            \
             H--I--J   <-- dev/br-2 (myown)

Now that we have the commits we want, we merely need to place the particular labels. Since one or more of these labels will point to commit G', it helps to redraw the above with H'-I'-J' on a row of its own:

                   H'-I'-J'  <-- copied (HEAD)
                  /
           E'-F'-G'
          /
A--B--C--D   <-- master (upstream)
    \
     E--F--G   <-- dev/br-1 (upstream)
            \
             H--I--J   <-- dev/br-2 (myown)

The labels we want moved are:

• dev/br1, maybe; it should point to G';
• upstream/dev/br1—but this is our copy of upstream's dev/br1, so we don't move it directly, we have the remote named upstream move theirs so that our Git updates our memory of their name;
• origin/dev/br1: this is just like upstream/dev/br1;
• dev/br2: this should point to J'; and
• myown/dev/br2 or maybe origin/dev/br2, but once again this is our Git's memory of some other repository's branch name, so we have to convince that other Git repository to move their name.

To move our own dev/br1, we can simply use git branch -f now:

git branch -f dev/br1 copied~3

for instance. Since the name copied selects commit J', and the ~3 suffix means "move back three first-parents", that will select commit G'. The -f means force and causes our Git to move our dev/br1.

To move upstream's dev/br1 and cause our upstream/dev/br1 to move, we now need to git push --force-with-lease or similar to upstream, which also assumes that we have permission (on whatever system hosts upstream: Git itself doesn't "do" permissions, but sites like GitHub and others do, for obvious reasons). The --force-with-lease tells our Git to verify that their dev/br1 still points where we expect; if we're sure that this will be the case, we can use plain --force. Either way the command is, or resembles:

git push --force-with-lease upstream dev/br1
git push --force-with-lease origin dev/br1

which uses the fact that we forced our br1 to point to G'.

The same process applies to making the various names point to H', except that now we can use the name copied:

git branch -f dev/br2 copied
git push --force-with-lease origin dev/br2

Once we have done all this, we can switch to dev/br2 or whatever and delete the extra branch name copied. It existed only so that we had a nice simple way to find commit H' after all the copying.

The key here is to understand that the names are largely irrelevant. All that actually matters are the commits, which are identified by their hash IDs. The trick is that we find the commits using the names, so that makes the names relevant.

Alternatives

If you like, you can still do this with two git rebase operations. Since things are relatively simple, we don't need the fancy --onto for the first one (but we will need it for the second):

git switch dev/br-1
git rebase upstream/master

This takes our starting point, which looks like this—note that I'm assuming things about the remote-tracking names this time:

A--B--C--D   <-- upstream/master
    \
     E--F--G   <-- dev/br-1 (HEAD), upstream/dev/br-1
            \
             H--I--J   <-- dev/br-2, origin/dev/br-2

and copies E-F-G to new-and-improved E'-F'-G', placing them after commit D, as named by upstream/master:

           E'-F'-G'  <-- dev/br-1 (HEAD)
          /
A--B--C--D   <-- upstream/master
    \
     E--F--G   <-- upstream/dev/br-1
            \
             H--I--J   <-- dev/br-2, origin/dev/br-2

Having made the three copies, git rebase yanked the name dev/br-1 off commit G and made it point to commit G' instead.

Now we will separately copy H-I-J:

git switch dev/br-2
git rebase --onto dev/br-1 upstream/dev/br-1

Here we needed the --onto to tell Git:

• Don't copy commits from upstream/dev/br-1 backwards, i.e., don't copy commits G and earlier, but that's not where we want to put the copies!
• Put the copies after commit G', i.e., the newly updated dev/br-1.

The result of this is:

                   H'-I'-J'  <-- dev/br-2 (HEAD)
                  /
           E'-F'-G'  <-- dev/br-1
          /
A--B--C--D   <-- upstream/master
    \
     E--F--G   <-- upstream/dev/br-1
            \
             H--I--J   <-- origin/dev/br-2

As with the first git rebase, Git did the copying of the commits found from the current branch name (i.e., J and back), excluding the commits we said not to copy (i.e., G and back), placing the copies wherever they go—this time that's separate from the "what not to copy" part—and then, having made the copies, git rebase yanked the name dev/br-2 to point to the final copied commit (J').

With the (local) repository's names now pointing to the copies, it's once again just a matter of using git push --force-with-lease or git push --force to get the other Git software, working with the two other repositories, to update their branch names, so that our own Git's memories in our remote-tracking names get updated.

(If you can't literally force-push to upstream or origin, you can still send the updated commits, e.g., via pull requests, but someone else will have to convince the other repositories to actually move their branch names to take in the new commits.)

Answer 2

WARNING: This is an extremely long answer as it includes every step's command, output and status. It is a log of how I achieved my goal, which was:

========

1. I forked an upstream, and made my own changes from its development branches (dev/br-1), in my own new branch (dev/br-2), and git push to my own repo.

   Situation at point#1:

   A---B---C---D  master (upstream)
       \
        E---F---G  dev/br-1 (upstream)
                \
                 H---I---J dev/br-2 (myown)
   

2. Now, upstream has advanced, both in its master and develop branches.

3. The way upstream advanced its develop branch is via rebase, i.e., all its own changes are put back on top after rebase.

   Situation at point#2&3:

   A---B---C---D  master (upstream)
               \
                 E'---F'---G'  dev/br-1 (upstream)
   

4. My local repo is gone with my old machine, and I need to pick-up/git-clone from my own repo to continue with my customization.

5. I want to rebase my changes in my dev/br-2 branch (which shares a common ancestor) on top of all the upstream changes.

I.e., the ultimate goal is to place myown

H---I---J dev/br-2

branch on top of the newly rebased G', in my own repo, after catching up with upstream. I.e., in my own repo, in the end, it should look like this:

A---B---C---D  rebased master (upstream)
            \
             E'---F'---G'  rebased dev/br-1 (upstream)
                       \
                        H'---I'---J' rebased dev/br-2

========

Solution:

Do it with three git rebase operations.

Again, The details of the preparation steps can be found at pastebin.com/Df8VbCp2, included below to make a full story:

$ cd /tmp

$ rm -rf upstream/ myfork/

$  mkdir upstream

$  cd upstream

# == prepare its `master` and `dev/br-1` branches

$  git init

seq 12 | tee file-a
git commit -am 'add file-a'

$  git checkout -b dev/br-1
Branch 'dev/br-1' set up to track local branch 'master' by rebasing.
Switched to a new branch 'dev/br-1'

sed -i 's/^1/7/' file-a

git commit -am 'update file-a'

seq 5 6 | tee -a file-a
git commit -am 'more updates'
seq 12 | tee file-b
git commit -am 'more updates'

 cd /tmp
 git clone upstream myfork
 # == prepare my own changes based on the `dev/br-1` branch into `dev/br-2`

$  git branch -avv
* master                  2f5eaaf [origin/master] add file-a
  remotes/origin/HEAD     -> origin/master
  remotes/origin/dev/br-1 0006c5e more updates
  remotes/origin/master   2f5eaaf add file-a

$  git checkout --track origin/dev/br-1
Branch 'dev/br-1' set up to track remote branch 'dev/br-1' from 'origin' by rebasing.
Switched to a new branch 'dev/br-1'

$ git checkout -b dev/br-2
Branch 'dev/br-2' set up to track local branch 'dev/br-1' by rebasing.
Switched to a new branch 'dev/br-2'

sed -i '/9/{ N; N; s/^.*$/3\n4/; }' file-a 

$ cat file-a
7
2
3
4
5
6
7
8
3
4
72
5
6

git commit -am 'update file-a in dev/br-2'

 cd /tmp/upstream
 # advance its `master` 

 sed -i 's/^5/55/' file-a
 git commit -am 'update file-a from main'

 # and its`dev/br-1` branches
 git checkout dev/br-1
 git rebase -X theirs master dev/br-1

$ cat file-a
7
2
3
4
55
6
7
8
9
70
71
72
5
6

# Now upstream has advanced, both in its master and its develop branches (via rebase)

Continuing from there -- Start from point#4, pick-up/git-clone from my own repo to continue with my customization, and then first rebase master, then br-1, then br-2, using

git rebase upstream/master

git switch dev/br-1
git rebase upstream/master

git switch dev/br-2
git rebase --onto dev/br-1 upstream/dev/br-1

Details:

# I need to pick-up from my own repo to continue with my customization.
 cd /tmp
 mv myfork myfork0
 git clone myfork0 myfork1
 cd myfork1
 git remote -v
 git remote add upstream /tmp/upstream
 git remote -v
 git fetch upstream

$ git log --all --decorate --oneline --graph
* 7aec18c (upstream/dev/br-1) more updates
* b83c3f8 more updates
* 4cf241f update file-a
* 200959c update file-a from main
* dc45a94 (upstream/master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (HEAD -> master, origin/master, origin/HEAD) add file-a

$ git rebase upstream/master
Successfully rebased and updated refs/heads/master.

$ git log --all --decorate --oneline --graph
* 7aec18c (upstream/dev/br-1) more updates
* b83c3f8 more updates
* 4cf241f update file-a
* 200959c update file-a from main
* dc45a94 (HEAD -> master, upstream/master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

$ git switch dev/br-1
fatal: 'dev/br-1' matched multiple (2) remote tracking branches
$ git checkout --track origin/dev/br-1
Branch 'dev/br-1' set up to track remote branch 'dev/br-1' from 'origin' by rebasing.
Switched to a new branch 'dev/br-1'

$ git log --all --decorate --oneline --graph
* 7aec18c (upstream/dev/br-1) more updates
* b83c3f8 more updates
* 4cf241f update file-a
* 200959c update file-a from main
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (HEAD -> dev/br-1, origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a


$ git rebase upstream/master
Successfully rebased and updated refs/heads/dev/br-1.

$ git log --all --decorate --oneline --graph
* 53d26a9 (HEAD -> dev/br-1) more updates
* 3ad83f2 more updates
* c18eab7 update file-a
| * 7aec18c (upstream/dev/br-1) more updates
| * b83c3f8 more updates
| * 4cf241f update file-a
| * 200959c update file-a from main
|/  
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a



$ git switch dev/br-2
Branch 'dev/br-2' set up to track remote branch 'dev/br-2' from 'origin' by rebasing.
Switched to a new branch 'dev/br-2'

$ git log --all --decorate --oneline --graph
* 53d26a9 (dev/br-1) more updates
* 3ad83f2 more updates
* c18eab7 update file-a
| * 7aec18c (upstream/dev/br-1) more updates
| * b83c3f8 more updates
| * 4cf241f update file-a
| * 200959c update file-a from main
|/  
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (HEAD -> dev/br-2, origin/dev/br-2) update file-a
| * 0006c5e (origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

$ git rebase --onto dev/br-1 upstream/dev/br-1
Auto-merging file-a
CONFLICT (content): Merge conflict in file-a
error: could not apply 85afa56... update file-a
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
. . .

# fix it, then
git add file-a

$ git rebase --continue
Successfully rebased and updated refs/heads/dev/br-2.

$ git log --all --decorate --oneline --graph
* 76d524a (HEAD -> dev/br-2) update file-a
* 53d26a9 (dev/br-1) more updates
* 3ad83f2 more updates
* c18eab7 update file-a
| * 7aec18c (upstream/dev/br-1) more updates
| * b83c3f8 more updates
| * 4cf241f update file-a
| * 200959c update file-a from main
|/  
* dc45a94 (upstream/master, master) update file-a from main
| * ce2a804 (origin/dev/br-2) update file-a
| * 0006c5e (origin/dev/br-1) more updates
| * cdee8bb more updates
| * 85afa56 update file-a
|/  
* 2f5eaaf (origin/master, origin/HEAD) add file-a

$ cat file-a
7
2
3
4
55
6
7
8
3
4
72
5
6