20230727_velero备份迁移kubernetes集群
一、velero安装
1# wget https://github.com/vmware-tanzu/velero/releases/download/v1.11.1/velero-v1.11.1-linux-amd64.tar.gz
2# tar -zxvf velero-v1.11.1-linux-amd64.tar.gz
3# cp -av velero /usr/bin/
4
5#velero -h
6# 启用命令补全
7# source <(velero completion bash)
8# velero completion bash > /etc/bash_completion.d/velero
1# cat > credentials-velero <<EOF
2[default]
3aws_access_key_id = admin
4aws_secret_access_key = P@ssw0rd
5EOF
6
7# velero install \
8 --provider aws \
9 --image velero/velero:v1.11.1 \
10 --plugins velero/velero-plugin-for-aws:v1.6.0 \
11 --bucket velero \
12 --secret-file ./credentials-velero \
13 --use-node-agent \
14 --use-volume-snapshots=false \
15 --namespace velero \
16 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.31.110:9000 \
17 --wait
18
19 --prefix /cce-test
二、卸载
1kubectl delete namespace/velero clusterrolebinding/velero
2kubectl delete crds -l component=velero
三、执行velero 备份与恢复
1# 不执行存储pvc内容备份
2velero backup create my-velero-backup-pvc --include-namespaces=foo
3
4# 通过配置--default-volumes-to-fs-backup 则会备份foo空间内存在存在pvc卷的容器。
5velero backup create my-velero-backup-pvc --include-namespaces=foo --default-volumes-to-fs-backup
6
7# 恢复的过程需要回复pvc卷，默认不加--restore-volumes也能够进行恢复
8velero restore create my-restore-without-pvc --from-backup=my-velero-backup-pvc --include-namespaces=foo --restore-volumes=true
四、备份策略
场景一、常规备份命名空间opp
1# 完成后会在OSS 生成目录及文件 /ack-test/backups/opp/*
2# velero backup create opp --include-namespaces opp --wait
3
4# 删除命名空间opp
5# kubectl delete namespace opp
6
7# 使用velero恢复命名空间opp
8# 完成后会在OSS 生成目录及文件 /ack-test/restores/opp-20221024163120/*
9# velero restore create --from-backup opp --wait
场景二、 集群迁移
1在源集群上进行备份，然后在目标集群上进行恢复
场景三、定时备份
1# 定时备份的名称：<schedule-name>-<timestamp>，如：test-ack-schedule-every-five-minute-20221024090654
2# 默认情况下保留30天
3# --schedule="0 1 * * *" 每日1点进行备份
4# --schedule="0 1 * * *" --tl 48h 每日1点进行备份，备份保留48小时
5# --schedule="@every 6h" 每6小时进行一次备份
6
7# 创建定时备份集
8# velero create schedule ack-test-schedule-every-five-minute --schedule="@every 5m"
9Schedule "ack-test-schedule-every-five-minute" created successfully.
10
11# 列出当前所有备份集
12# velero schedule get ack-test-schedule-every-five-minute
13NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR
14ack-test-schedule-every-five-minute Enabled 2022-10-24 17:06:54 +0800 CST @every 5m 0s 2m ago <none>
15
16# 查看定时备份集
17# velero schedule describe ack-test-schedule-every-five-minute
18
19# 删除定时备份集
20# velero schedule delete ack-test-schedule-every-five-minute
21
22
23# 使用cron表达式备份
24# velero schedule create nginx-daily --schedule="0 1 * * *" --include-namespaces nginx-example
25
26# 使用一些非标准的速记 cron 表达式
27# velero schedule create nginx-daily --schedule="@daily" --include-namespaces nginx-example
28
29# 手动触发定时任务
30# velero backup create --from-schedule nginx-daily
备份操作
1#备份集群ingress-nginx namespace下资源:
2velero backup create ingress-nginx-backup --include-namespaces ingress-nginx
3
4#查看备份结果
5velero backup describe ingress-nginx-backup
6velero backup logs ingress-nginx-backup
7
8#删除备份
9velero delete backup ingress-nginx-backup
10
11#备份非ingress-nginx和test命名空间下的资源：
12velero backup create k8s-full-test-backup --exclude-namespaces ingress-nginx,test
13
14#备份特定资源类型
15velero backup create kube-system-backup --include-resources pod,secret
16
17#--confirm 直接删除备份，无需确认：
18velero backup delete kube-system-backup --confirm
19
20#备份带pv pod
21velero backup create pvc-backup --snapshot-volumes --include-namespaces test-velero
恢复操作
1#从backup创建restore
2velero restore create ${RESTORE_NAME} --from-backup ${BACKUP_NAME}
3
4# 从backup创建restore，restore默认名为 ${BACKUP_NAME}-<timestamp>
5velero restore create --from-backup ${BACKUP_NAME}
6
7# 从schedule最新一次的backup创建restore
8velero restore create --from-schedule ${SCHEDULE_NAME}
9
10# 指定backup中的某些资源创建restore
11velero restore create --from-backup backup-2 --include-resources pod,secret
12
13# 恢复集群所有备份，（对已经存在的服务不会覆盖）
14velero restore create --from-backup all-ns-backup
15
16# 仅恢复default nginx-example命名空间
17velero restore create --from-backup all-ns-backup --include-namespaces default,nginx-example
18
19# 将test-velero 命名空间资源恢复到test-velero-1下面
20velero restore create restore-for-test --from-backup everyday-1-20210203131802 --namespace-mappings test-velero:test-velero-1
四、关于velero 备份PVC卷数据的问题。
需要注意，早期如果需要使用restic 的方式进行 pvc卷的备份时，则需要在velero install的过程进行手动指定安装选项。
如：
1velero install \ 2 --provider aws \ 3 --image velero/velero:v1.11.1 \ 4 --plugins velero/velero-plugin-for-aws:v1.6.0 \ 5 --bucket velero \ 6 --secret-file ./credentials-velero \ 7 --use-node-agent \ 8 --use-volume-snapshots=false \ 9 --namespace velero \ 10 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.31.110:9000 \ 11 --wait \ 12 --use-restic \ 13 --default-volumes-to-restic–ues-restic 与–defult-volumes-to-restic 选项， 在velero 1.11.1当中已经没有该配置选项了。
在新的1.11.1+版本当中，会创建一个backuprepositories对象，该对象会记录restic 针对已经备份pvc卷数据的对象存储位置，如果冒然去吧，/bucket/restic/ 文件夹在对象存储上删除，在进行恢复过程当中，或进行二次--default-volumes-to-fs-backuppvc备份， 则会提示报错提示如下：
1# velero backup get my-velero-backup-pvc-1
2NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
3my-velero-backup-pvc-1 InProgress 0 0 2023-07-28 13:27:21 +0800 CST 29d default <none>
4[root@3113-nfs velero]# velero backup describe my-velero-backup-pvc-1
5Name: my-velero-backup-pvc-1
6Namespace: velero
7Labels: velero.io/storage-location=default
8Annotations: velero.io/source-cluster-k8s-gitversion=v1.25.6
9 velero.io/source-cluster-k8s-major-version=1
10 velero.io/source-cluster-k8s-minor-version=25
11
12Phase: PartiallyFailed (run `velero backup logs my-velero-backup-pvc-1` for more information)
13
14
15Errors:
16 Velero: name: /nginx-57bb8cd95c-5w7x6 error: /pod volume backup failed: error creating uploader: failed to connect repository: error running command=restic snapshots --repo=s3:http://192.168.31.110:9000/velero/restic/foo --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic --latest=1, stdout=, stderr=Fatal: unable to open config file: Stat: The specified key does not exist.
17Is there a repository at the following location?
18s3:http://192.168.31.110:9000/velero/restic/foo
19: exit status 1
20 name: /nginx-57bb8cd95c-5w7x6 error: /pod volume backup failed: error creating uploader: failed to connect repository: error running command=restic snapshots --repo=s3:http://192.168.31.110:9000/velero/restic/foo --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic --latest=1, stdout=, stderr=Fatal: unable to open config file: Stat: The specified key does not exist.
21Is there a repository at the following location?
22s3:http://192.168.31.110:9000/velero/restic/foo
23: exit status 1
24
25...略...
26
27
28# kubectl logs -n velero velero-7f7d4d8f94-gwd7t
29
30time="2023-07-28T05:27:31Z" level=error msg="Error backing up item" backup=velero/my-velero-backup-pvc-1 error="pod volume backup failed: error creating uploader: failed to connect repository: error running command=restic snapshots --repo=s3:http://192.168.31.110:9000/velero/restic/foo --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic --latest=1, stdout=, stderr=Fatal: unable to open config file: Stat: The specified key does not exist.\nIs there a repository at the following location?\ns3:http://192.168.31.110:9000/velero/restic/foo\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/podvolume/backupper.go:255" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:448" name=nginx-with-loadaware-847db647f-fnlhv
31
32time="2023-07-28T05:27:46Z" level=error msg="Error checking repository for stale locks" backupRepo=velero/foo-default-restic-fpfb9 error="error running command=restic unlock --repo=s3:http://192.168.31.110:9000/velero/restic/foo --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic, stdout=, stderr=Fatal: unable to open config file: Stat: The specified key does not exist.\nIs there a repository at the following location?\ns3:http://192.168.31.110:9000/velero/restic/foo\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/repository/restic/repository.go:123" error.function="github.com/vmware-tanzu/velero/pkg/repository/restic.(*RepositoryService).exec" logSource="pkg/controller/backup_repository_controller.go:182"
经过测试，在进行pvc卷备份过程当中，对于从未进行备份的命名空间第一次会创建如下对象：
1# kubectl get backuprepositories.velero.io -n velero
2NAME AGE REPOSITORY TYPE
3default-default-restic-5rqjn 6m5s restic
4foo-default-restic-tsrql 7m37s restic
5
6# kubectl get backuprepositories.velero.io -n velero foo-default-restic-tsrql -o yaml
7apiVersion: velero.io/v1
8kind: BackupRepository
9metadata:
10 creationTimestamp: "2023-07-28T05:36:56Z"
11 generateName: foo-default-restic-
12 generation: 3
13 labels:
14 velero.io/repository-type: restic
15 velero.io/storage-location: default
16 velero.io/volume-namespace: foo
17 name: foo-default-restic-tsrql
18 namespace: velero
19 resourceVersion: "132229791"
20 uid: 58f5f08e-7c65-4832-b2c6-615cbb55a07e
21spec:
22 backupStorageLocation: default
23 maintenanceFrequency: 168h0m0s
24 repositoryType: restic
25 resticIdentifier: s3:http://192.168.31.110:9000/velero/restic/foo
26 volumeNamespace: foo
27status:
28 lastMaintenanceTime: "2023-07-28T05:36:58Z"
29 phase: Ready
如果需要处理该报错，需要删除 backuprepositories 对象，删除之后会再次创建。
参考：
https://www.cnblogs.com/wubolive/p/17345716.html 利用Velero对K8S备份还原与集群迁移实战
- 原文作者：Kid
- 原文链接：https://shuanglu.life/post/20230727_velero%E5%A4%87%E4%BB%BD%E8%BF%81%E7%A7%BBkubernetes%E9%9B%86%E7%BE%A4/
- 版权声明：本作品采用知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议进行许可，非商业转载请注明出处（作者，原文链接），商业转载请联系作者获得授权。
