ForgeOps

Secrets Rotation

Introduction

Secrets rotation is the process of updating or replacing sensitive information stored as Kubernetes secrets. Secrets rotation is crucial for maintaining strong security and mitigating risks of unauthorized access or data breaches.

In a multi-component system, such as a ForgeOps deployment, each component interacts with others using secrets. Therefore, it’s important to consider dependencies among components and perform secrets rotation while maintaining consistent interaction among dependent components.

The forgeops command includes the rotate sub-command to enable ds-env-secrets and ds-passwords rotation consistently. You can rotate other ForgeOps required secrets also with due consideration to the impact on dependent components and downtime.

Performing secrets and passwords rotation

This section describes how to rotate secrets and password in ForgeOps deployments. The steps for rotating each secret are mentioned separately for easier understanding and usage.

Rotating ds-env-secrets

The ds-env-secrets controls access to DS from AM and IDM, and would normally cause a downtime when rotated. To avoid such a downtime, the forgeops rotate command creates old-ds-env-secrets temporarily to contain old secrets.

In ForgeOps release 2025.2.1, the DS image was built to accommodate multiple passwords. This enables secrets rotation with no downtime.

For deployments using DS images from 2025.1 or earlier

If you are using the DS image from the 2025.1 release or earlier, then perform these steps to enable multiple passwords in DS.

  1. In your terminal window, set up environment variables to get the password and connection string (DSPASS and CONN_STR):

    $ export DSPASS=$(kubectl get secret ds-passwords -n my_ns -o yaml | yq '.data["dirmanager.pw"]' | tr -d '"' | base64 -d -i -)

$ export CONN_STR="--hostname localhost --port 4444 --bindDn uid=admin --trustAll --no-prompt --bindPassword $DSPASS"

  2. Set up DS pods to enable multiple passwords:

    $ kubectl exec -it ds-cts-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Default Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

$ kubectl exec -it ds-cts-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Root Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

$ kubectl exec -it ds-idrepo-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Default Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

$ kubectl exec -it ds-idrepo-0 — bin/dsconfig set-password-policy-prop \
  set-password-policy-prop --policy-name "Root Password Policy" \
  --set allow-multiple-password-values:true $CONN_STR

To rotate ds-env-secrets, run the forgeops rotate --namespace my_ns ds-env-secrets command.

The command prompts you to perform steps to complete rotation of ds-env-secrets.

Rotating ds-passwords

To rotate ds-passwords, run the forgeops rotate --namespace my_ns ds-passwords command. The command prompts you to perform steps to complete rotation of ds-passwords.

You must restart the DS pods to update the admin user password because that password is set on DS pod startup. This could also require restarting some services instead of redeploying components.

At the end of its successful run, the forgeops rotate command prompts the user to:

  • Delete the temporary secrets.

  • Remove the old passwords.

Rotating amster secret

Impact

This secret is specific to Amster and doesn’t cause a downtime.

Procedure

  1. Trigger renewal of Kubernetes secret:

    1. When using secret agent:

      $ kubectl delete secret amster

    2. When using secrets generator:

      $ kubectl annotate secrets amster secret-generator.v1.mittwald.de/regenerate=amster

  2. Rolling restart AM pods to pick up new amster secret:

    $ kubectl rollout restart deployment am -n my-ns

  3. Refresh amster job to verify amster can access AM:

    1. Delete amster job to allow redeployment:

      $ kubectl delete job amster -n my-ns

    2. Deploy platform changes (amster)

      1. For Helm:

        $ helm upgrade -i identity-platform --repo https://ForgeRock.github.io/forgeops/ \
 --version my-prod-version \
 --values /path/to/forgeops/helm/my-env/values.yaml

      2. For Kustomize:

        $ forgeops apply amster --env-name my-env

  4. Ensure that the amster import process has completed successfully:

    $ kubectl logs -f amster-pod -n my-ns

Rotating am-env-secrets

Impact

AM depends on this secret for authentication. Therefore, during the short time between restarting AM and rerunning Amster, requests that need authentication could fail.

Procedure

  1. Trigger renewal of Kubernetes secret:

    1. When using secret agent:

      $ kubectl delete secret am-env-secrets

    2. When using secrets generator:

      $ kubectl annotate secrets idm-env-secrets secret-generator.v1.mittwald.de/regenerate=am-env-secrets

  2. Rolling restart AM pods to pick up new amster secret:

    $ kubectl rollout restart deployment am -n my-ns

  3. Reinitiate the amster job:

    1. Delete the amster job to allow redeployment:

      $ kubectl delete job amster -n my-ns

    2. Redeploy amster:

      1. For Helm:

        $ helm upgrade -i identity-platform --repo https://ForgeRock.github.io/forgeops/ \
  --values /path/to/forgeops/helm/my-env/values.yaml

      2. For Kustomize:

        $ forgeops apply amster --env-name my-env

  4. Ensure that the amster import process has completed successfully:

    $ kubectl logs -f amster-pod -n my-ns

  5. Retrieve the new password for amadmin user to log in to the platform:

    $ forgeops info | grep amadmin

  6. Log in to the platform with new amadmin password to verify the platform is up and running.

Rotating amster-env-secrets

Impact

In the very short time between restarting IDM and Amster importing necessary data, the platform isn’t accessible. Amster takes a few seconds to import data.

Procedure

  1. Trigger renewal of Kubernetes secret:

    1. When using secret agent:

      $ kubectl delete secret amster-env-secrets

    2. When using secrets generator:

      $ kubectl annotate secrets amster-env-secrets secret-generator.v1.mittwald.de/regenerate=amster-env-secrets

  2. Rolling restart IDM pods to get the new amster secret:

    $ kubectl rollout restart deployment idm -n my-ns

  3. Rerun the amster job to import the new secrets:

    1. Delete the amster job to allow redeployment:

      $ kubectl delete job amster -n my-ns

    2. Redeploy Amster:

      1. When using Helm to deploy:

        $ helm upgrade -i identity-platform --repo https://ForgeRock.github.io/forgeops/ --values /path/to/custom/values.yaml

      2. When using Kustomize deploy:

        $ forgeops apply --env-name my-env amster

Rotating idm-env-secrets

Procedure

  1. Trigger renewal of Kubernetes secret:

    1. When using secret agent:

      $ kubectl delete secret idm-env-secrets

    2. When using secrets generator:

      $ kubectl annotate secrets idm-env-secrets secret-generator.v1.mittwald.de/regenerate=OPENIDM_ADMIN_PASSWORD

  2. Rolling restart IDM pods:

    $ kubectl rollout restart deployment idm

  3. Check pods have come up:

    $ kubectl get pods -l app.kubernetes.io/component=idm -n my-ns

Rotating ds-ssl-keypair

It’s not advisable to rotate this secret. If you rotate this secret DS data replication will fail until all the DS pods are restarted.
Procedure

  1. Delete the ds-ssl-keypair secret:

    $ kubectl delete secret ds-ssl-keypair -n my-ns

  2. Check that the secret is recreated:

    $ kubectl get secret ds-ssl-keypair -n my-ns

  3. Rolling restart ds-cts pods to pick up new secret:

    $ kubectl rollout restart sts ds-cts -n my-ns

  4. Rolling restart ds-idrepo pods to pick up new secret:

    $ kubectl rollout restart sts ds-idrepo -n my-ns

  5. Rolling restart AM pods to pick up new secret:

    $ kubectl rollout restart deployment am -n my-ns

  6. Rolling restart IDM pods to pick up new secret:

    $ kubectl rollout restart deployment idm -n my-ns

  7. Check pods to ensure they have come back up:

    $ kubectl get pods -l app.kubernetes.io/component=ds-cts -n my-ns
$ kubectl get pods -l app.kubernetes.io/component=ds-idrepo -n my-ns
$ kubectl get pods -l app.kubernetes.io/component=am -n my-ns
$ kubectl get pods -l app.kubernetes.io/component=idm -n my-ns

Rotating am-passwords

Impact

Rotating am-passwords doesn’t necessitate a down-time. am-passwords is used only in ForgeOps deployments that use secret agent for secrets management. It’s not relevant for deployments that use secrets generator.

Procedure

  1. Trigger renewal of Kubernetes secret:

    $ kubectl delete secret am-passwords
$ kubectl delete secret am-keystore

  2. Delete the AM pod:

    $ kubectl delete pod am-wxyz-abcd

  3. Recreate keystore to use the new secret.

Rotating keystore-create

Impact

You can rotate keystore-create only when using secret generator and provisioning a new keystore.

Procedure

  1. Trigger renewal of Kubernetes secret:

    $ kubectl annotate secrets idm-env-secrets secret-generator.v1.mittwald.de/regenerate=keystore-create

  2. Delete the keystore-create job:

    $ kubectl delete job keystore-create

  3. Redeploy platform products to redeploy the keystore-create job:

    $ kubectl rollout restart deployment am
$ kubectl rollout restart deployment idm

  4. Verify if the pods have come up:

    $ kubectl get pods -l app.kubernetes.io/component=am -n my-ns

Rotating ds-master-keypair

Do not rotate ds-master-keypair because it’s used in DS data backups. The same secret is required to decrypt data when you need to restore from backups. If you must rotate these secrets, then restart backups and discard the previous backups because they can’t be used for restoring directory data.