Secrets Rotation

Introduction

Secrets rotation is the process of updating or replacing sensitive information stored as Kubernetes secrets. Secrets rotation is crucial for maintaining strong security and mitigating risks of unauthorized access or data breaches.

In a multi-component system, such as a ForgeOps deployment, each component interacts with others using secrets. Therefore, it’s important to consider dependencies among components and perform secrets rotation appropriately.

The forgeops command includes the rotate sub-command to enable secrets rotation consistently.

In ForgeOps release 2025.2.1, the DS image was built to accommodate multiple passwords. This enables secrets rotation with no downtime.

For deployments using DS images from 2025.1 or earlier

If you are using the DS image from the 2025.1 release or earlier, then perform these steps to enable multiple passwords in DS.

In your terminal window, set up environment variables to get the password and connection string (DSPASS and CONN_STR):

$ export DSPASS=$(kubectl get secret ds-passwords -n my_ns -o yaml | yq '.data["dirmanager.pw"]' | tr -d '"' | base64 -d -i -)

$ export CONN_STR="--hostname localhost --port 4444 --bindDn uid=admin --trustAll --no-prompt --bindPassword $DSPASS"

Set up DS pods to enable multiple passwords:

$ kubectl exec -it ds-cts-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Default Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

$ kubectl exec -it ds-cts-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Root Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

$ kubectl exec -it ds-idrepo-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Default Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

$ kubectl exec -it ds-idrepo-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Root Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

Performing rotation

Use the forgeops rotate command to perform rotation of secrets and passwords stored in ds-env-secrets and ds-passwords. ds-env-secrets and ds-passwords control access to DS from AM and IDM, and cause downtime when they are rotated. To avoid such a downtime, the forgeops rotate command creates temporary secrets that contain the old secrets and passwords.

For example, when you run forgeops rotate --namespace my_ns ds-env-secrets, it creates a new temporary secret called old-ds-env-secrets.

Similarly, when you run forgeops rotate --namespace my_ns ds-passwords, it creates old-ds-passwords temporarily.

After creating the temporary secrets or passwords, the script prompts for the next steps to follow, such as:

Run the ds-set-passwords job through Helm or Kustomize and update the passwords.
Redeploy AM with Helm or Kustomize used in your deployment.

Applying these changes depends on the secret management utility - secret-agent, secret-generator, or another option - used in the environment.

Rotating ds-set-secrets

Run the forgeops rotate --namespace my_ns ds-env-secrets command to rotate ds-set-secrets. The command prompts you for the steps to perform to complete the rotation, such as:

When using Helm to deploy

Set ds_set_passwords.force to true in your values.yaml file.
Run the helm upgrade -i ... command to redeploy.
Verify that the ds-set-passwords job completes successfully.
Restart AM pods if necessary.

$ kubectl rollout restart deployment am

When using Kustomize to deploy

Apply the ds-set-passwords job.

$ kubectl apply -k /path/to/kustomize/overlay/my_env/ds-set-passwords
Restart AM

$ kubectl rollout restart deployment am

Rotating ds-passwords: Run forgeops rotate --namespace my_ns ds-passwords to rotate ds-passwords. The command prompts you for the steps to perform to complete the rotation. Note that you must restart the DS pods to update the admin user password, as that password is set on DS pod startup. This could also require restarting some services instead of running Helm or Kustomize to redeploy components.

At the end of its successful run, the forgeops rotate command prompts user to:

Delete the temporary secrets.
Remove the old passwords.