btrfs send / receive on incremental folders that rotate

Question

I am using incremental backups using rsnapshot combined with a custom cmd_cp and cmd_rm to make use of btrfs snapshots, this procudes multiple daily btrfs subvolumes:

.sync
daily.0
daily.1
daily.2

.sync is the folder that gets synced to using SSH from the server I backup. When completed this happens:

mv daily.2/ daily.3/
mv daily.1/ daily.2/
mv daily.0/ daily.1/
rsnapshot_cp_btrfs -al .sync daily.0

The cp command translates into btrfs subvolume snapshot -r .sync daily.0.

This all works great. But now I want to sync all backups to an other server too, so I have a full mirror of all backups. This sync should always work, even if it would be synced a week after (maybe due SSH connection issues).

Normally syncing would be easy using btrfs send and receive using parent snapshots as described on the wiki: https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Doing_it_by_hand.2C_step_by_step

I imagined a loop that just sends all daily folders and maintaining the old backup for parent reference.

But in this case daily.0 moves to daily.1, 1 to 2, and so on. So this would not work.

I could send a simple mv to the remote server, but I can't trust this since on any errors, the folder structures would not be proper a day later. I want a true mirror, but making use of the btrfs tools.

Did anyone managed a similar situation or know the best way to clone all subvolumes to the other server?

Big thanks!

Answer 1

I solved it! I created a bash script that syncs all snapshots with a date in the name to the remote server. The date is subtracted from btrfs subvolume show.

So daily.0 can become 2021-09-20-08-44-46 on the remote.

I sync backwards. daily.30 first. daily.0 last. This way I can pass the proper parent to btrfs send. E.g.: btrfs send -p daily.30 daily.29.

If the date named snapshot exists on the remote, I check with btrfs subvolume show whether it was properly synced. If not, I delete the remote subvolume/snapshot and re-sync. If it was already synced, I will skip the sync. A proper synced subvolume/snapshot has a Received UUID and readonly flag.

After syncing I compare all snapshot names of the remote to what was just synced. The difference will deleted (thus old snapshots).

I might share the code in the future when it's all been stable for a long run. For now I hope the above information will help others!