milvus-backup restore very slow , why and how to improve performance?

Question

I try to backup and restore milvus 2.3.3 with milvus-backup 0.4 , making backup of collection with 160M entities seems pretty fast in 30min , but restore take more then 9h , please, advice if there is options to improve , the command I am using to restore is:

 milvus-backup restore -n mycollection --restore_index

P.S. I see milvus during restore is almost not using resource , only the pod with the backup-tool seems to do something:

k top pods -n milvus --sort-by=cpu
NAME                                   CPU(cores)   MEMORY(bytes)
milvus-backup-mycluster               1002m          330Mi
mycluster-datanode-5788666898-jb47f    777m          855Mi
mycluster-rootcoord-687cf4cb9-wp6n8     76m          132Mi
mycluster-querycoord-c47b99b8c-hflsc    40m          137Mi
mycluster-datacoord-84657c87df-zf9ld    27m          207Mi
mycluster-proxy-78d487487b-v4x22        20m          127Mi
mycluster-proxy-78d487487b-ntsjk        20m          134Mi
mycluster-etcd-2                        20m          148Mi
mycluster-etcd-1                        19m          137Mi
mycluster-etcd-0                        19m          134Mi
mycluster-pulsar-broker-0               19m          1465Mi

Answer 1

You can increase the datanode number and change parallelism config in backup.yaml.

backup:
  parallelism: 
    # Collection level parallelism to restore
    restoreCollection: 2

Answer 2

The recovery speed depends on bulk insert performance. During recovery, increase the number of datanode to improve bulk insert speed