ForgeOps

Backup and restore using volume snapshots

Kubernetes volume snapshots provide a standardized way to create copies of persistent volumes at a point in time without creating new volumes. Backing up your directory data with volume snapshots lets you perform rapid recovery from the last snapshot point. Volume snapshot backups also facilitate testing by letting you initialize DS with sample data.

In the CDM, the DS data, change log, and configuration are stored in the same persistent volume. This ensures the volume snapshot captures DS data and changelog together.

The backup and restore procedure using volume snapshots described here is meant for use in ForgeOps release 7.4 deployment environments where ds-operator is not used.

Backup

Set up backup

The Kustomize overlays necessary to set up volume snapshots of the CDM deployed to the prod namespace are provided in the kustomize/overlay/ds-snapshot directory of the forgeops repository. These overlays are not handled by the forgeops command.

When enabled, the default setup of volume snapshot takes snapshot of the data-ds-idrepo-0 and data-ds-cts-0 PVCs once a day.

To enable volume snapshot of DS data from the my-namespace namespace using the default settings, perform the following steps:

  1. In a terminal window, change to the ds-snapshot subdirectory under the kustomize/overlay directory:

    $ cd /path/to/forgeops/kustomize/overlay/ds-snapshot
  2. Copy the content of the prod directory to a new directory with the name of the namespace where you have deployed CDM:

    $ cp -rp ./prod ./my-namespace
  3. Change to the my-namespace directory.

  4. Edit the /rbac/namespace.yaml file and change the last line to specify the namespace in which CDM has been deployed.

  5. Set up the configuration map and enable volume snapshot backup using the kubectl apply command:

    $ kubectl apply --kustomize configmap --namespace my-namespace
    $ kubectl apply --kustomize rbac --namespace my-namespace
    $ kubectl apply --kustomize idrepo --namespace my-namespace
  6. Optionally, if you want to back up the cts as well, then run the following:

    $ kubectl apply --kustomize cts --namespace my-namespace
  7. View the volume snapshots that are available for restore, using this command:

    $ kubectl get volumesnapshots --namespace my-namespace
    
    NAME                               READYTOUSE   SOURCEPVC          SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS       SNAPSHOTCONTENT
              CREATIONTIME   AGE
    ds-idrepo-snapshot-20231117-1320   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-be3f4a44-cfb2-4f68-aa2b-60902
    bb44192   3h29m          3h29m
    ds-idrepo-snapshot-20231117-1330   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-7bcf6779-382d-40e3-9c9f-edf31
    c54768e   3h19m          3h19m
    ds-idrepo-snapshot-20231117-1340   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-c9c88332-ad05-4880-bda7-48616
    ec13579   3h9m           3h9m
    ds-idrepo-snapshot-20231117-1401   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-1f3f4ce9-0083-447f-9803-f6b45
    e03ac27   167m           167m
    ds-idrepo-snapshot-20231117-1412   true         data-ds-idrepo-0                           100Gi         ds-snapshot-class   snapcontent-4c39c095-0891-4da8-ae61-fac78
    c7147ff   156m           156m

Customize backup schedule

When enabled, volume snapshots are created once every day by default, and purged after three days. To modify the default schedule and purge delay, edit the schedule.yaml file in cts and idrepo directories, and run the kubectl apply command.

Examples for scheduling snapshots
  • To schedule snapshot twice a day, at 12:00 noon and midnight:

    ...
      spec:
        schedule: "0 0/12 * * *"
    ...

  • To schedule snapshot every 8 hours:

    ...
      spec:
        schedule: "0 */8 * * *"
    ...

Examples for purging schedule
  • To schedule purge after 4 days:

    ...
             env:
               - name: PURGE_DELAY
                 value: "-4 day"

  • To schedule purge after a week:

    ...
             env:
               - name: PURGE_DELAY
                 value: "-7 day"

Restore from volume snapshot

ForgeOps team provides the snapshot-restore.sh script to restore the DS instances in the CDM. This script restores a DS instance from the latest available snapshot, by default.

The snapshot-restore.sh script requires the JQ utility to manage JSON files used in restore operations. You must install JQ before using the snapshot-restore.sh script.

There are two options when using the snapshot-restore.sh script to restore a DS from a volume snapshot:

  • Full: Use the full option to fully restore a DS instance from a volume snapshot. In this option, the DS is scaled down to 0 pods before restoring data. The data is restored to an existing PVC from a snapshot. This operation requires downtime.

  • Selective: Use the selective option to restore a select portion of DS data from volume snapshot. The selective restore creates a new temporary DS instance with a new DS pod. You can selectively export from the temporary DS pod and import into your functional DS instance. After restoring data, you can clean up the temporary resources.

The snapshot-restore.sh command is available in the bin directory of the forgeops repository. To learn more about the snapshot-restore.sh command, use snapshot-restore.sh --help command to learn more about the command and its options.

Restore examples

Trial run without actually restoring DS data
  1. In a terminal window, change to /path/to/forgeops/bin directory.

  2. Set your Kubernetes context to the correct cluster and namespace.

  3. Run the snapshot-restore.sh command with --dryrun option:

    $ ./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo
    
    ./snapshot-restore.sh --dryrun --namespace my-namespace full idrepo
    /usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts-restore.json -n my-namespace
    /usr/local/bin/kubectl delete pvc data-ds-idrepo-0 -n my-namespace
    /usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/data-ds-idrepo-0.json -n my-namespace
    /usr/local/bin/kubectl apply -f /tmp/snapshot-restore-idrepo.20231121T23:03:15Z/sts.json -n my-namespace
Full restore of the idrepo instance from the latest available volume snapshot
  1. In a terminal window, change to /path/to/forgeops/bin directory.

  2. Set your Kubernetes context to the correct cluster and namespace.

  3. Get a list of available volume snapshots:

    $ kubectl get volumesnapshots --namespace my-namespace
  4. Restore full DS instance:

    $ ./snapshot-restore.sh --namespace my-namespace full idrepo
  5. Verify that DS data has been restored.

Selective restore from a specific volume snapshot and storing data in a user-defined storage path
  1. In a terminal window, change to /path/to/forgeops/bin directory.

  2. Set your Kubernetes context to the correct cluster and namespace.

  3. View the available volume snapshots, using this command:

    $ kubectl get volumesnapshots --namespace my-namespace
  4. Perform selective restore trial run:

    $ ./snapshot-restore.sh --dryrun --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo
    
    VolumeSnapshot ds-idrepo-snapshot-20231121-2250 is ready to use
    /usr/local/bin/kubectl apply -f /tmp/ds-rest/sts-restore.json -n my-namespace
    /usr/local/bin/kubectl apply -f /tmp/ds-rest/svc.json -n my-namespace
  5. Perform selective restore using a specific snapshot:

    $ ./snapshot-restore.sh --path /tmp/ds-restore --snapshot ds-idrepo-snapshot-20231121-2250 --namespace my-namespace selective idrepo
    
    statefulset.apps/ds-idrepo-restore created
    service/ds-idrepo configured
  6. Verify a new ds-idrepo-restore-0 pod is created:

    $ kubectl get pods
    NAME                          READY   STATUS      RESTARTS   AGE
    admin-ui-656db67f54-2brbf     1/1     Running     0          3h17m
    am-7fffff59fd-mkks5           1/1     Running     0          107m
    amster-hgkv9                  0/1     Completed   0          3h18m
    ds-idrepo-0                   1/1     Running     0          39m
    ds-idrepo-restore-0           1/1     Running     0          2m40s
    end-user-ui-df49f79d4-n4q54   1/1     Running     0          3h17m
    idm-fc88578bf-lqcdj           1/1     Running     0          3h18m
    login-ui-5945d48fc6-ljxw2     1/1     Running     0          3h17m

    The ds-idrepo-restore-0 pod is temporary and not to be used as a complete DS instance. You can export required data from the temporary pod, and import data into your functional DS instance.

    The following sample commands are meant to be examples and are not to be used in production.

  7. Connect to the ds-idrepo-restore-0 pod and run the export-ldif command, for example:

    $ kubectl exec ds-idrepo-restore-0 -it — bash
    $ export-ldif \
     --includeBranch dc=example,dc=com \
     --backendId userData \
     --ldifFile /path/to/DS/ldif/my-export.ldif \
     --offline
  8. Copy the exported LDIF file from ds-idrepo-restore-0 pod to a local folder:

    $ kubectl cp ds-idrepo-restore-0:/path/to/DS/ldif/my-export.ldif /path/to/local/destination
  9. Copy the exported file from the local folder to the ds-idrepo-0 pod:

    $ kubectl cp /path/to/local/destination/my-export.ldif ds-idrepo-0:/path/to/DS/ldif
  10. Import data into the ds-idrepo instance:

    $ kubectl exec ds-idrepo-0 -it — bash
    $ import-ldif --includeBranch dc=example,dc=com --backendId userData --ldifFile ds-idrepo-0:/path/to/DS/ldif/my-export.ldif
  11. Clean up resources from selective restore:

    $ ./snapshot-restore.sh clean idrepo
    
    statefulset.apps "ds-idrepo-restore" deleted
    persistentvolumeclaim "data-ds-idrepo-restore-0" deleted