A Complete Guide to Use the Milvus Backup Tool with wxd
Introduction
Milvus Backup is a powerful tool designed for backing up and restoring Milvus data, offering both CLI and API interfaces to suit a wide range of use cases. Whether you're managing a large-scale Milvus deployment or simply ensuring data safety, this tool simplifies backup operations, helping you maintain your Milvus instance with ease and confidence.
Milvus Backup works by reading collection metadata and segments from the source Milvus instance to create a backup. It then copies the collection data from the root directory of the source Milvus instance and stores it in a designated backup location.
When it's time to restore, Milvus Backup uses the collection metadata and segment information from the backup to create a new collection in the target Milvus instance. It then transfers the backup data from the backup directory to the root directory of the target instance, ensuring a seamless data restoration process.
Prerequisite:
Before start using Milvus Backup, ensure that
- The operating system is CentOS 7.5+ or Ubuntu LTS 18.04+
- Go version is 1.20.2 or later
-
Bucket Credentials: You’ll need valid credentials for both the Milvus instance and the backup store bucket to successfully perform backup and restore operations.
-
Version Compatibility: This tool supports backup and restoration across different versions of Milvus 2.x, allowing you to back up data from a lower version and restore it to a higher version.
-
Minimal Downtime: The Milvus cluster remains fully functional during the backup and restore process, ensuring no disruption to operations.
-
Flexible Restoration: Collection metadata and indexes can be restored using a bulk insert. Additionally, indexes can be restored by providing an extra parameter in the restoration request, making the process more customisable.
-
Cross Storage Backup: Data is read from the source storage and written to a different storage through the Milvus-backup service. Such as, S3 -> local, S3 a -> S3 b.
- MinIO Bucket
- IBM Cloud
- AWS S3
- Google Cloud Platform (GCP) via JSON credentials
- Private Azure storage
Before you can start backing up or restoring data, you'll need to set up your environment. You can either download the compiled binary or build from source.
1. Download the compiled binary
- Go to the milvus-backup releases page, where you can find all official releases. Select the appropriate version for your operating system:
For macOS, choose milvus-backup_Darwin_arm64.tar.gz or milvus-backup_Darwin_x86_64.tar.gz
For Linux, choose milvus-backup_Darwin_x86_64.tar.gz or milvus-backup_Linux_x86_64.tar.gz
- Extract the tar file to your preferred directory
2. Use homebrew to install on Mac
brew install zilliztech/tap/milvus-backup
3. Compile from the source
Do as follows:
git clone git@github.com:zilliztech/milvus-backup.git
go get
go build
Prepare Configuration File
-
Download the sample configuration file and customise it according to your requirements.
-
Create a folder named configs in the same directory where the Milvus Backup binary is located.
-
Place the customised backup.yaml configuration file inside the configs folder.
Your folder structure should look like this:
workspace
├── milvus-backup
└── configs
└── backup.yaml
Available Commands
Usage:
milvus-backup [flags]
milvus-backup [command]
Available Commands:
check check if the connects is right.
create create a backup.
delete delete backup by name.
get get subcommand get backup by name.
help Help about any command
list shows all backup in the cluster.
restore restore a backup.
server start milvus-backup RESTAPI server.
Flags:
--config string to specify config YAML file of milvus-backup (default "backup.yaml")
-h, --help help for milvus-backup
Use "milvus-backup [command] --help" for more information about a command.
Create Backup Operation
Usage:
milvus-backup create [flags]
Flags:
-n, --name string backup name will be automatically generated if not specified.
-c, --colls string specify the collection names to back up, using commas to separate multiple collections.
-d, --databases string databases to backup
-a, --database_collections string databases and collections to back up, in JSON format: `{"db1":["c1", "c2"],"db2":[]}`.
-f, --force force backup will skip the flush process. Ensure that the data has been saved to disk before using this option.
--meta_only only backup collection meta instead of data
-h, --help displays help for creating a backup.
Example: Backing up a collection named "promo" with the backup named "promoBackup"
./milvus-backup create -c promo --config ./configs/backup.yaml -n promoBackup
Restore a Backup Operation
Usage:
milvus-backup restore [flags]
Flags:
-n, --name string backup name to restore
-c, --collections string collectionNames to restore
-s, --suffix string add a suffix to collection name to restore
-r, --rename string rename collections to new names, format: db1.collection1:db2.collection1_new,db1.collection2:db2.collection2_new
-d, --databases string databases to restore, if not set, restore all databases
-a, --database_collections string databases and collections to restore, json format: {"db1":["c1", "c2"],"db2":[]}
--meta_only if true, restore meta only
--restore_index if true, restore index
--use_auto_index if true, replace vector index with autoindex
--drop_exist_collection if true, drop existing target collection before create
--drop_exist_index if true, drop existing index of target collection before create
--skip_create_collection if true, will skip collection, use when collection exist, restore index or data
-h, --help displays help for creating a backup.
Example: Restoring the "promoBackup" named backup, including the index.
./milvus-backup restore --restore_index --config ./configs/backup.yaml -n promoBackup -s _recover
Below is a sample success log for restoring a backup:
Read more:
#watsonx.data
