--- title: Backup and restore using volume snapshots description: Kubernetes volume snapshots provide a standardized way to create copies of persistent volumes at a point in time without creating new volumes. Backing up your directory data with volume snapshots lets you perform rapid recovery from the last snapshot point. Volume snapshot backups also facilitate testing by letting you initialize DS with sample data. component: forgeops version: 7.4 page_id: forgeops::how-to/backup/snapshots canonical_url: https://docs.pingidentity.com/forgeops/7.4/how-to/backup/snapshots.html keywords: ["Backup & Restore", "Volume Snapshot"] section_ids: backup: Backup set_up_backup: Set up backup customize_backup_schedule: Customize backup schedule restore_from_volume_snapshot: Restore from volume snapshot restore_examples: Restore examples --- # Backup and restore using volume snapshots Kubernetes [volume snapshots](https://kubernetes.io/docs/concepts/storage/volume-snapshots) provide a standardized way to create copies of persistent volumes at a point in time without creating new volumes. Backing up your directory data with volume snapshots lets you perform rapid recovery from the last snapshot point. Volume snapshot backups also facilitate testing by letting you initialize DS with sample data. In the CDM, the DS data, change log, and configuration are stored in the same persistent volume. This ensures the volume snapshot captures DS data and changelog together. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The backup and restore procedure using volume snapshots described here is meant for use in ForgeOps release 7.4 deployment environments where `ds-operator` is **not** used. | ## Backup ### Set up backup The Kustomize overlays necessary to set up volume snapshots of the CDM deployed to the `prod` namespace are provided in the kustomize/overlay/ds-snapshot directory of the `forgeops` repository. These overlays are not handled by the forgeops command. When enabled, the default setup of volume snapshot takes snapshot of the `data-ds-idrepo-0` and `data-ds-cts-0` PVCs once a day. To enable volume snapshot of DS data from the my-namespace namespace using the default settings, perform the following steps: 1. In a terminal window, change to the ds-snapshot subdirectory under the kustomize/overlay directory: ``` $ cd /path/to/forgeops/kustomize/overlay/ds-snapshot ``` 2. Copy the content of the `prod` directory to a new directory with the name of the namespace where you have deployed CDM: ``` $ cp -rp ./prod ./my-namespace ``` 3. Change to the my-namespace directory. 4. Edit the /rbac/namespace.yaml file and change the last line to specify the namespace in which CDM has been deployed. 5. Set up the configuration map and enable volume snapshot backup using the kubectl apply command: ``` $ kubectl apply --kustomize configmap --namespace my-namespace $ kubectl apply --kustomize rbac --namespace my-namespace $ kubectl apply --kustomize idrepo --namespace my-namespace ``` 6. Optionally, if you want to back up the `cts` as well, then run the following: ``` $ kubectl apply --kustomize cts --namespace my-namespace ``` 7. View the volume snapshots that are available for restore, using this command: ``` $ kubectl get volumesnapshots --namespace my-namespace NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE ds-idrepo-snapshot-20231117-1320 true data-ds-idrepo-0 100Gi ds-snapshot-class snapcontent-be3f4a44-cfb2-4f68-aa2b-60902 bb44192 3h29m 3h29m ds-idrepo-snapshot-20231117-1330 true data-ds-idrepo-0 100Gi ds-snapshot-class snapcontent-7bcf6779-382d-40e3-9c9f-edf31 c54768e 3h19m 3h19m ds-idrepo-snapshot-20231117-1340 true data-ds-idrepo-0 100Gi ds-snapshot-class snapcontent-c9c88332-ad05-4880-bda7-48616 ec13579 3h9m 3h9m ds-idrepo-snapshot-20231117-1401 true data-ds-idrepo-0 100Gi ds-snapshot-class snapcontent-1f3f4ce9-0083-447f-9803-f6b45 e03ac27 167m 167m ds-idrepo-snapshot-20231117-1412 true data-ds-idrepo-0 100Gi ds-snapshot-class snapcontent-4c39c095-0891-4da8-ae61-fac78 c7147ff 156m 156m ``` ### Customize backup schedule When enabled, volume snapshots are created once every day by default, and purged after three days. To modify the default schedule and purge delay, edit the schedule.yaml file in `cts` and `idrepo` directories, and run the kubectl apply command. * Examples for scheduling snapshots * To schedule snapshot twice a day, at 12:00 noon and midnight: ``` ... spec: schedule: "0 0/12 * * *" ... ``` * To schedule snapshot every 8 hours: ``` ... spec: schedule: "0 */8 * * *" ... ``` * Examples for purging schedule * To schedule purge after 4 days: ``` ... env: - name: PURGE_DELAY value: "-4 day" ``` * To schedule purge after a week: ``` ... env: - name: PURGE_DELAY value: "-7 day" ``` ## Restore from volume snapshot ForgeOps team provides the snapshot-restore.sh script to restore the DS instances in the CDM. This script restores a DS instance from the latest available snapshot, by default. | | | | - | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The snapshot-restore.sh script requires the `JQ` utility to manage JSON files used in restore operations. You must install `JQ` before using the snapshot-restore.sh script. | There are two options when using the snapshot-restore.sh script to restore a DS from a volume snapshot: * Full: Use the full option to fully restore a DS instance from a volume snapshot. In this option, the DS is scaled down to 0 pods before restoring data. The data is restored to an existing PVC from a snapshot. This operation requires downtime. * Selective: Use the selective option to restore a select portion of DS data from volume snapshot. The selective restore creates a new temporary DS instance with a new DS pod. You can selectively export from the temporary DS pod and import into your functional DS instance. After restoring data, you can clean up the temporary resources. The snapshot-restore.sh command is available in the `bin` directory of the `forgeops` repository. To learn more about the snapshot-restore.sh command, use snapshot-restore.sh --help command to learn more about the command and its options. ### Restore examples * Trial run without actually restoring DS data 1. In a terminal window, change to /path/to/forgeops/bin directory. 2. Set your Kubernetes context to the correct cluster and namespace. 3. Run the snapshot-restore.sh command with `--dryrun` option: ``` $ ./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo ./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo /usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts-restore.json -n my-namespace /usr/local/bin/kubectl delete pvc data-ds-idrepo-0 -n my-namespace /usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/data-ds-idrepo-0.json -n my-namespace /usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts.json -n my-namespace ``` * Full restore of the `idrepo` instance from the latest available volume snapshot 1. In a terminal window, change to /path/to/forgeops/bin directory. 2. Set your Kubernetes context to the correct cluster and namespace. 3. Get a list of available volume snapshots: ``` $ kubectl get volumesnapshots --namespace my-namespace ``` 4. Restore full DS instance: ``` $ ./snapshot-restore.sh --namespace my-namespace full idrepo ``` 5. Verify that DS data has been restored. * Selective restore from a specific volume snapshot and storing data in a user-defined storage path 1. In a terminal window, change to /path/to/forgeops/bin directory. 2. Set your Kubernetes context to the correct cluster and namespace. 3. View the available volume snapshots, using this command: ``` $ kubectl get volumesnapshots --namespace my-namespace ``` 4. Perform selective restore trial run: ``` $ ./snapshot-restore.sh --dryrun --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo VolumeSnapshot ds-idrepo-snapshot-20231121-2250 is ready to use /usr/local/bin/kubectl apply -f /tmp/ds-rest/sts-restore.json -n my-namespace /usr/local/bin/kubectl apply -f /tmp/ds-rest/svc.json -n my-namespace ``` 5. Perform selective restore using a specific snapshot: ``` $ ./snapshot-restore.sh --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo statefulset.apps/ds-idrepo-restore created service/ds-idrepo configured ``` 6. Verify a new `ds-idrepo-restore-0` pod is created: ``` $ kubectl get pods NAME READY STATUS RESTARTS AGE admin-ui-656db67f54-2brbf 1/1 Running 0 3h17m am-7fffff59fd-mkks5 1/1 Running 0 107m amster-hgkv9 0/1 Completed 0 3h18m ds-idrepo-0 1/1 Running 0 39m ds-idrepo-restore-0 1/1 Running 0 2m40s end-user-ui-df49f79d4-n4q54 1/1 Running 0 3h17m idm-fc88578bf-lqcdj 1/1 Running 0 3h18m login-ui-5945d48fc6-ljxw2 1/1 Running 0 3h17m ``` | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | The `ds-idrepo-restore-0` pod is temporary and not to be used as a complete DS instance. You can export required data from the temporary pod, and import data into your functional DS instance. | | | | | - | -------------------------------------------------------------------------------------------- | | | The following sample commands are meant to be examples and are not to be used in production. | 7. Connect to the `ds-idrepo-restore-0` pod and run the export-ldif command, for example: ``` $ kubectl exec ds-idrepo-restore-0 -it — bash $ export-ldif \ --includeBranch dc=example,dc=com \ --backendId userData \ --ldifFile /path/to/DS/ldif/my-export.ldif \ --offline ``` 8. Copy the exported LDIF file from `ds-idrepo-restore-0` pod to a local folder: ``` $ kubectl cp ds-idrepo-restore-0:/path/to/DS/ldif/my-export.ldif /path/to/local/destination ``` 9. Copy the exported file from the local folder to the `ds-idrepo-0` pod: ``` $ kubectl cp /path/to/local/destination/my-export.ldif ds-idrepo-0:/path/to/DS/ldif ``` 10. Import data into the `ds-idrepo` instance: ``` $ kubectl exec ds-idrepo-0 -it — bash $ import-ldif --includeBranch dc=example,dc=com --backendId userData --ldifFile ds-idrepo-0:/path/to/DS/ldif/my-export.ldif ``` 11. Clean up resources from selective restore: ``` $ ./snapshot-restore.sh clean idrepo statefulset.apps "ds-idrepo-restore" deleted persistentvolumeclaim "data-ds-idrepo-restore-0" deleted ```