Access Management 7.2.2

Core Token Service (CTS)

These pages describe how to configure the Core Token Service, and how to manage the tokens stored in it.

Deployment architecture

Choose the best architecture for your deployment.
Configure a dedicated CTS store

Prepare a DS instance to host a dedicated CTS token store.
Manage tokens

Learn how to encrypt and/or compress tokens.
Manage expired CTS tokens

Configure how CTS tokens are pruned after they reach their maximum time-to-live.

ForgeRock® Identity Platform serves as the basis for our simple and comprehensive Identity and Access Management solution. We help our customers deepen their relationships with their customers, and improve the productivity and connectivity of their employees and partners. For more information about ForgeRock and about the platform, see https://www.forgerock.com.

CTS overview

AM’s Core Token Service (CTS) provides generalized, persistent, and highly available storage for sessions and tokens used by AM. AM uses CTS as the authoritative source for server-side sessions and caches these sessions in its memory heap to improve performance.(1)

CTS supports session high availability, which lets AM manage a session as long as one of the AM servers in a clustered deployment is available. After a user has successfully authenticated, AM creates a server-side session. Any AM instance that is configured to use the same CTS can retrieve the session and allow access to it. The user does not need to log in again unless the entire deployment goes down.(2)

CTS provides storage for the following:

  • Server-side sessions and server-side authentications sessions

  • Session denylist (if enabled for client-side sessions)

  • Authentication session allowlist (if enabled for client-side sessions)

  • SAML v2.0-related data (if enabled for Security Token Service token validation and cancellation)

  • OAuth 2.0 and UMA 2.0 server-side tokens, and OAuth 2.0 client-side token denylist

  • Push notification data during authentication

  • Site-wide notification, such as log out or session termination notifications.

The CTS token store is created at installation time in the external configuration store.

In general, the Core Token Service causes more volatile replication traffic due to the possibility of short-lived entries compared to regular configuration data. To handle the data volatility, ForgeRock recommends that you configure an additional DS server, separate from your configuration store, to isolate session and token information.

CTS configuration recommendations

CTS helps your deployment avoid single points of failure by providing high availability to sessions and tokens stored in its token store. To reduce the impact of any given failure, consider the following recommendations:

  • Configure separate CTS stores for high volumes. If you require a higher-level performance threshold, you may want to move the CTS token storage to one or more dedicated systems rather than sharing your configuration store, as CTS generally causes much more replication traffic than less volatile configuration data. The CTS token store is the primary source for session tokens and will experience both high read and write activity depending on session usage. Dedicated external CTS stores provide an extra level of control over the amount of global replication that is occurring.

  • Isolate different stores. CTS entries are large, around 5KB, but are short-lived, whereas configuration data is static and long-lived. User entries are more dynamic than configuration data but much less volatile than CTS data. Therefore, isolating the user, configuration, and CTS data from AM into separate stores permit different tuning and storage settings per token store type.

    Configuration with separate identity repository and CTS token stores.
    Figure 1. Isolate the datastores

  • Tune DS servers. To improve performance, ensure that you have properly-sized directory servers for your external CTS stores. In addition, you can enable token compression as discussed in Manage CTS tokens. When enabled, token compression reduces load requirements on the network connection between token stores in exchange for processing time-compressing tokens.

  • Consider dedicated replication servers. Once configured, the DS server replicates CTS data transmitted from AM servers to connected DS servers. The amount of replication traffic can be significant, especially if replication proceeds over a WAN. You can limit this replication traffic by separating DS instances into directory and replication servers as seen in Core Token Service For Global Session Failover. For more information on how this is done with DS, see the DS documentation on Standalone Servers.

(1) Prior to AM 5, the authoritative source for sessions not stored in the client was the memory heap of AM’s web container.

(2) Prior to AM 5, session high availability, formerly referred to as session failover, was optional. Starting with AM 5, session high availability is the default behavior in AM and cannot be disabled.

Deployment architectures

You can deploy CTS token stores in a number of deployment architectures, depending on your system requirements:

Affinity CTS token store

For high availability in the same datacenter.
Site CTS token store

For high availability across datacenters.

CTS affinity deployment

In an affinity deployment, AM balances LDAP requests across one or more directory servers. AM always routes LDAP requests for a specific CTS token to the same directory server. This prevents attempts to read a token that has been written to another directory server, but not replicated yet. Affinity deployments are well suited for deployments with many AM servers.

Use AM’s Connection String(s) property on the AM admin UI to configure server affinity without a load balancer. For more information on the Connection String(s) property, see External Store Configuration.

Depicts an Affinity Configuration for CTS.
Figure 2. A CTS affinity deployment

Learn more about CTS affinity deployments in the Knowledge Base article Best practice for using Core Token Service (CTS) Affinity based load balancing in PingAM.

The connection strings to the data or identity stores are static and not hot-swappable. This means that, if you expand or contract your DS affinity deployment, AM will not detect the change.

To work around this, either:

  • Manually add or remove the instances from the connection string and restart AM or the container where it runs.

  • Configure a DS proxy in front of the DS instances to distribute data across multiple DS shards, and configure the proxy’s URL in the connection string.

CTS site deployment

CTS supports uninterrupted session availability in deployments with multiple sites if all sites use the same global underlying CTS store replicated across all sites. If an entire site fails or becomes unavailable, AM servers in another site can detect the failure of the site’s load balancer and attempt to use sessions from the global Core Token Service.

In the event of a failure, client applications can connect to an AM server in an active data center:

Shows a globally replicated Core Token Service with two data centers
Figure 3. Core token service for global session failover

For more information on global session availability for the CTS, see Replication in the Directory Services documentation.

Configure CTS token stores

The following table summarizes the high-level tasks you must perform to configure a new instance of the CTS token store:

Task Resources

Prepare a DS Server

Prepare the DS schema for CTS data.

Install and configure Directory Services for CTS data

Configure AM to Use the New CTS Token Store

Configuring a new CTS data store does not migrate existing data from the old store to the new one. In most cases, this will mean that users will need to log in again so that AM stores their sessions and tokens in the new token store.

Configure the CTS

Test Session High-Availability

For this test, you need to have a site with more than one instance. The idea is to log in a user into the first instance, shut it down, and check that the session is still available to the second instance.

Test session availability

Install and configure Directory Services for CTS data

Directory Services 6.5 added support for setup profiles to simplify initial configuration. Using a setup profile creates the backend, schema, bind user, and indexes required for use with CTS data.

  1. To install DS using a setup profile, follow the steps in Install DS for AM CTS in the Directory Services documentation.

  2. Share the CTS store certificate with the AM container to prepare for TLS/LDAPS. The CTS store should communicate over secure connections for security reasons.

    DS 7 and later requires secure connections by default; therefore, share its certificate with the AM container before continuing.

    Share the DS certificate with AM

    • On the DS host, export the DS CA certificate.

      DS uses a deployment ID and password to generate a CA key pair. Learn more in Deployment IDs.

      Use the dskeymgr command to export the CA certificate:

      $ /path/to/opendj/bin/dskeymgr \
export-ca-cert \
--deploymentId $DEPLOYMENT_ID \
--deploymentIdPassword password \
--outputFile /path/to/ca-cert.pem

    • Copy the ca-cert.pem file to an accessible location on the AM host.

    • Import the DS certificate into the AM truststore:

      $ keytool \
-importcert \
-file /path/to/ca-cert.pem \
-keystore /path/to/openam/security/keystores/truststore

    Learn more about configuring AM’s truststore in Prepare the truststore.

  3. Proceed to configuring the CTS store in AM. See Configure the CTS.

    The bind DN of the service account to use when configuring the CTS store in AM is uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens.

Configure the CTS

This procedure assumes that the AM instances communicate with a single CTS instance (or load balancer), which has an FQDN of cts.example.com and is running on port 1636.

If AM cannot access the CTS token store, you will be unable to log in to the AM admin UI.

Back up your deployment before making any changes to your CTS token store configuration.

Perform the following steps to configure an external CTS token store:

  1. In the AM admin UI, go to Configure > Server Defaults, and click CTS.

  2. On the CTS Token Store tab, configure the parameters as follows:

    CTS Token Store Parameters
    Parameter Value Notes

    Store Mode

    External Token Store

    Root Suffix

    ou=famrecords,ou=openam-session,ou=tokens

    Max Connections

    17

    For production, this value needs to be tuned. Consider 2n+1, where n=4, 5, 6, and so on. For example, try setting this to 17, 33, 65, and test performance under load.

  3. On the External Store Configuration tab, configure the parameters as follows:

    External Store Configuration Parameters
    Parameter Value Notes

    SSL/TLS Enabled

    True

    When connecting to DS 7, or a DS server in production mode, enable secure connections. When you enable SSL/TLS, make sure the AM server can trust the DS server certificate and that the certificate matches the CTS store FQDN.

    Connection String(s)

    cts.example.com:1636

    Use the LDAPS port. Alternatively, use the LDAP port with StartTLS, as in cts.example.com:1389.

    Login ID

    uid=openam_cts,ou=admins,ou=famrecords,ou=openam-session,ou=tokens

    The bind DN of the service account AM uses to connect to the directory service.

    Password

    5up35tr0ng

    Use a strong bind password for production systems.

    Heartbeat

    10

    For production, this value needs to be tuned.

  4. Save your work.

  5. Restart AM or the web container where it runs for the changes to take effect.

Test session availability

To test session availability, use two browsers: Chrome and Firefox. You can use any two browser types, or run the browsers in incognito mode. You can also view tokens using an LDAP browser.

  1. In Chrome, log in to the second AM instance with the amAdmin user, click the realm, and click Sessions.

  2. In Firefox, log in to the first AM instance with a test user.

  3. In Chrome, verify that the test user exists in the first AM instance’s session list and not in the second instance.

  4. Shut down the first AM instance.

  5. In Firefox, rewrite the URL to point to the second AM instance.

    If successful, the browser should not prompt for login.

  6. Confirm the session is still available.

    In Chrome, list the sessions on the second instance, the test user’s session should be present.

  7. Restart the first AM instance to complete the testing.

Manage CTS tokens

You can configure AM to encrypt or compress CTS tokens as they are stored in the token store. The following properties, disabled by default, are associated with token encryption and compression:

com.sun.identity.session.repository.enableEncryption

Supports encryption of CTS tokens. Default: false.

com.sun.identity.session.repository.enableCompression

Enables GZip-based compression of CTS tokens. Default: false.

com.sun.identity.session.repository.enableAttributeCompression

Supports compression over and above the GZip-based compression of CTS tokens. Default: false.

Compression can undermine the security of encryption. You should evaluate this threat, according to your use case, before you enable compression and encryption.

Configure AM to encrypt and compress CTS tokens for storage

When encryption or compression properties are changed, all previous tokens in the LDAP store will be unreadable; thus, invalidating any user’s sessions. As a result, the user will be required to log in again.

  1. Go to Configure > Server Defaults > Advanced.

  2. Find the property you want to enable in the Property Name column.

  3. Replace the false value with true in the Property Value column.

  4. Click Save Changes.

  5. Enable the same property on every AM instance within the site.

    Failure to do so may cause unexpected issues storing and reading tokens across the environment.

  6. Restart the AM servers for the changes to take effect.

Configuring the CTS to encrypt and store tokens incurs a performance penalty for AM. If you need to encrypt the stored tokens in your environment, consider configuring the CTS token store DS instance to encrypt the data instead. For more information about encrypting a DS instance, see the Data encryption in the Directory Services documentation.

CTS backups and DS replication purge delay

Replication is the process of copying updates between directory servers to help all servers converge to identical copies of directory, token, session, SAML v2.0, and OAuth 2.0 data. DS uses advanced data replication methods to ensure that directory services remain available in the event of a server crash or network interruption.

The historical information needed to resolve the latest changes is periodically purged to prevent it from becoming an unmanageable size. The age at which the information is purged is known as the replication-purge-delay.

With CTS, the default replication-purge-delay for DS is 3 days. Unless you have configured a separate DS server for CTS data, you may have to balance the needs for backups, the requirements for replication, disk space, and different useful lifetimes for CTS tokens and other DS data. Adjustments may be required. For example, to set the replication-purge-delay period to four hours, use the following command:

$ /path/to/opendj/bin/dsconfig set-synchronization-provider-prop \
--provider-name "Multimaster Synchronization" \
--set replication-purge-delay:4h \
--hostname 'cts.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePasswordFile /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--no-prompt

At this point, you need to understand whether CTS data backups are important in your deployment. Session, SAML v2.0, and OAuth 2.0 token data is often short-lived. In some deployments, the worst-case scenario is that users have to log in again.

If CTS data backups are important in your deployment, note that DS backups that are older than the replication-purge-delay are useless and must be discarded. You can use the DS dsbackup command to schedule backups. For more information, see Backup and Restore in the ForgeRock Directory Services documentation.

If you adjust the time periods associated with replication-purge-delay and backups, you must back up more frequently, so that the change log records required to restore data are not lost.

Manage expired CTS tokens

Tokens in the CTS store have a limited time to live, after which they expire and must be pruned. By default, AM manages expired tokens by using its own reaper. You can, however, delegate this task to DS.

Although both solutions work equally well, using DS to manage token expiration instead of AM’s reaper frees up resources in AM servers that can then be used for policy or authorization requests.

The following topics explain the two methods of managing expired CTS tokens in detail.

Manage expired tokens with the AM CTS reaper

When an AM server modifies a token in the CTS token store, it takes responsibility for managing that token when it expires. Each AM server maintains a local cache of expired tokens to delete, and when to delete them.

Using the local reaper cache reduces the number of searches to the CTS store to determine expired tokens to delete. This improves overall cluster performance. Servers still search the CTS store for expired tokens occasionally, to ensure expired tokens aren’t missed when a server in the cluster goes down.

The AM CTS reaper is enabled by default and requires no extra configuration. If you’ve configured the DS expiration and deletion feature instead, and then want to re-enable the AM CTS reaper, read Re-enable the AM reaper.

Manage expired tokens using DS

If you disable the AM CTS reaper, AM relies on DS capabilities to delete expired tokens.

DS doesn’t reap its contents by default–you must configure DS to expire and delete entries. Entry expiration and deletion uses an ordering index to find CTS tokens that have reached their time to live then expire and delete them.

Before you configure DS to manage CTS token expiration, consider the following points:

  • DS doesn’t replicate entry deletion across servers. You must make sure all CTS store replicas are configured in the same way. For details, refer to Entry expiration in the DS documentation.

  • Disabling the AM reaper impacts session-related functionality, such as sending notifications about session expiration or timeouts to agents.

    When you completely disable the AM reaper, the following session functionality becomes unavailable:

    • Implementations of org.forgerock.openam.cts.reaper.TokenDeletionStrategy, responsible for deleting expiring tokens.

      Custom logic for specific token types (unrelated to simply deleting tokens) no longer works.

    • Implementations of org.forgerock.openam.cts.continuous.watching.ContinuousListener and org.forgerock.openam.cts.continuous.watching.ContinuousWatcher no longer receive deletion notifications in case of session timeout.

      Active logout is not impacted.

    • Code built on top of com.iplanet.dpro.session.service.SessionEventListener no longer receives IDLE_TIMEOUT or MAX_TIMEOUT events. This affects:

      • Session timeout monitoring

      • Session timeout auditing

      • Session timeout logging

      • Session timeout notifications for agents (PLL, WebSockets)

      • Timeout handlers configured through the openam-session-timeout-handler-list

      • Session web hooks registered during authentication

    • Code built on top of com.iplanet.dpro.session.watchers.listeners.SessionDeletionListener no longer receives notifications when the session idle or maximum lifetime is exceeded (common usage includes caches).

    • Listeners built on top of com.sun.identity.plugin.session.SessionListener no longer receive notifications on active logout or when the maximum session time is reached (idle timeout is ignored).

    If you don’t need these session capabilities, refer to Let DS manage CTS tokens (ignore session capabilities).

    If you do need these session capabilities, refer to Let DS manage CTS tokens (retain session capabilities).

A common CTS token management strategy is to configure AM to reap session tokens only, and to use DS to manage non-session tokens. This configuration lets you retain session-related functionality, while benefiting from DS’s token management capabilities.

If your environment requires it, you can also configure the AM reaper to manage different subsets of tokens other than session tokens.

Let DS manage CTS tokens (ignore session capabilities)

These steps assume you don’t need the session capabilities impacted by disabling the AM CTS reaper. They configure DS to manage expiration and deletion of all CTS tokens in your environment.

  1. Perform one of the following steps, depending on whether you’re installing a new DS instance for CTS, or modifying an existing instance:

    • If you’re installing a new DS instance, use the CTS setup profile.

      Choose the option "DS manages all token expiration".

    • If you have an existing DS instance that you’re using as a CTS store, configure the DS entry expiration and deletion feature for the coreTokenExpirationDate attribute.

      This attribute is already indexed, but you must enable the TTL-related properties: 
      $ /path/to/opendj/bin/dsconfig set-backend-index-prop \
--hostname 'ds.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--backend-name amCts \
--index-name coreTokenExpirationDate \
--set ttl-enabled:true \
--set ttl-age:10s \
--no-prompt

      DS doesn’t replicate entry deletion across servers. You must configure all CTS store replicas in the same way. Learn more in Entry expiration in the DS documentation.

  2. Disable the AM CTS reaper.

    Go to Configure > Server Defaults > Advanced and set the org.forgerock.services.cts.store.reaper.enabled property to false.

    This change doesn’t require a server restart. Learn more about this advanced server property in Advanced properties.

    Perform this step on all AM servers in your deployment.

Let DS manage CTS tokens (retain session capabilities)

These steps assume you need the session capabilities that would be impacted by disabling the AM CTS reaper. They configure AM to reap SESSION tokens, while using DS capabilities to manage other tokens.

  1. Perform one of the following steps, depending on whether you’re installing a new DS instance for CTS, or modifying an existing instance:

    • If you’re installing a new DS instance, use the CTS setup profile.

      Choose the option "AM reaper manages only SESSION token expiration".

    • If you’re modifying an existing DS instance, follow these steps:

      1. If you have existing tokens in the CTS store, copy the value of the coreTokenExpirationDate attribute to the coreTokenTtlDate for all existing tokens except for SESSION tokens.

        AM will manage the SESSION tokens separately.

        If you don’t perform this step, neither the AM reaper nor the DS expiration feature will delete these tokens. They’ll remain in the CTS store until you remove them manually.

      2. Create an ordering index for the coreTokenTtlDate attribute:

        $ /path/to/opendj/bin/dsconfig create-backend-index \
--hostname 'ds.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--backend-name amCts \
--index-name coreTokenTtlDate \
--set index-type:ordering \
--no-prompt

      3. Rebuild the new index using the rebuild-index command:

        $ /path/to/opendj/bin/rebuild-index \
--hostname 'ds.example.com' \
--port 4444 \
--useSsl \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--baseDN "dc=cts,dc=example,dc=com" \
--index coreTokenTtlDate

      4. Configure entry expiration and deletion for the coreTokenTtlDate index you created in the previous step:

        $ /path/to/opendj/bin/dsconfig set-backend-index-prop \
--hostname 'ds.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--index-name coreTokenTtlDate \
--backend-name amCts \
--set ttl-enabled:true \
--set ttl-age:10s \
--no-prompt
      DS doesn’t replicate deletion of entries across servers. You must configure all CTS store replicas in the same way.

      For more information, refer to Entry expiration in the DS documentation.

  2. In the AM admin UI, go to Configure > Server Defaults > Advanced and configure the following advanced server properties for each AM server that shares the CTS store cluster:

    • Set org.forgerock.services.cts.store.ttlsupport.enabled to true.

    • Set org.forgerock.services.cts.store.ttlsupport.exclusionlist to SESSION.

    • Set org.forgerock.services.cts.store.reaper.enabled to true.

    The changes are effective immediately. Learn more about these advanced server properties in Advanced properties.

Re-enable the AM reaper

Follow these steps if you previously enabled the DS expiration and deletion feature but want to disable it and re-enable the AM reaper.

  1. If you enabled the DS expiration and deletion feature following the steps in Let DS manage CTS tokens (ignore session capabilities), follow these steps to enable the AM CTS reaper for all tokens:

    • Disable DS entry expiration and deletion for the coreTokenExpirationDate index:

      $ /path/to/opendj/bin/dsconfig set-backend-index-prop \
--hostname 'ds.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--backend-name amCts \
--index-name coreTokenExpirationDate \
--set ttl-enabled:false \
--no-prompt
      Don’t delete the index. Configure all CTS store replicas in the same way.

    • Re-enable the AM CTS reaper:

      On each AM server in your deployment, go to Configure > Server Defaults > Advanced and set the org.forgerock.services.cts.store.reaper.enabled property to true.

  2. If you enabled the DS expiration and deletion feature following the steps in Let DS manage CTS tokens (retain session capabilities), follow these steps to enable the AM CTS reaper for all tokens:

    • Disable the DS entry expiration and deletion feature for the coreTokenTtlDate index. For example:

      $ /path/to/opendj/bin/dsconfig set-backend-index-prop \
--hostname 'ds.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--index-name coreTokenTtlDate \
--backend-name amCts \
--set ttl-enabled:false
      Configure all CTS store replicas in the same way.

    • Delete the index created for the coreTokenTtlDate attribute. For example:

      $ /path/to/opendj/bin/dsconfig delete-backend-index \
--hostname 'ds.example.com' \
--port 4444 \
--usePkcs12TrustStore /path/to/opendj/config/keystore \
--trustStorePassword:file /path/to/opendj/config/keystore.pin \
--bindDN uid=admin \
--bindPassword str0ngAdm1nPa55word \
--backend-name amCts \
--index-name coreTokenTtlDate \
--no-prompt
      Configure all CTS store replicas in the same way.

    • Go to Configure > Server Defaults > Advanced and configure the following advanced server properties in all of the AM servers sharing the CTS store cluster:

      • Set org.forgerock.services.cts.store.ttlsupport.enabled to false.

      • Remove the org.forgerock.services.cts.store.ttlsupport.exclusionlist property from the configuration.

      • Set org.forgerock.services.cts.store.reaper.enabled to true.

    The changes are effective immediately. For more information about these advanced server properties, see Advanced properties.

Tune the CTS

There are several tuning considerations that you can make for the efficient processing of your CTS token store:

Task Resources

Configure the reaper cache size

The reaper process prunes tokens in the store that have reached their maximum time-to-live.

Reaper cache size

Manage the reaper search size

In addition to the cache, the reaper also uses a search to find expired tokens. You can tune the maximum amount of tokens that the reaper can find and delete in a single invocation.

Reaper search size

Manage the CTS queue size and timeout

Every create, update, and delete operation sent to the CTS token store is placed in an asynchronous queue before being handled. When the queue is full, new operations are blocked until they get into the queue or time out.

Queue size and timeout

Disable virtual attributes

To improve CTS throughput, disable unnecessary virtual DS attributes.

Virtual attributes

Change the OAuth 2.0 Storage Scheme

If the CTS token store was created on a version of AM earlier than 6.5, configure the grant-set scheme to improve the performance of OAuth 2.0-related operations.

OAuth 2.0 CTS storage scheme

Reaper cache size

The size of the AM reaper cache is controlled by the org.forgerock.services.cts.reaper.cache.size advanced property. The default size is 500000 tokens.

If an AM server is under sustained heavy load, the reaper cache may reach capacity, causing degraded performance due to the additional slower searches of the CTS store. If the reaper cache is full, messages are logged in the Session debug log, such as the following:

The CTS token reaper cache is full. This will result in degraded performance.
You should increase the cache size by setting the advanced server property
'org.forgerock.services.cts.reaper.cache.size' to a number higher than 500000.

If this debug message appears frequently in the debug logs, increase the value of the org.forgerock.services.cts.reaper.cache.size property. To alter the value, in the AM admin UI, go to Configure > Server Defaults > Advanced, and add the property and increased value to the list.

Increasing the size of the reaper cache causes higher memory usage on the AM server. If a cache of the default size of 500000 entries is nearly full, the server memory used could be up to approximately 100 megabytes.

Tune the AM reaper cache size only if the org.forgerock.services.cts.store.reaper.enabled advanced server property is set to true.

Reaper search size

The reaper uses its cache to ensure that expired tokens are removed from the CTS token store, but it also has a built-in mechanism to search for expired tokens that may have not been purged. This could happen, for example, due to an AM instance crash.

When the reaper searches for expired tokens, DS returns a page of records that then the reaper will delete.

To configure the maximum amount of records that DS will return, use the org.forgerock.services.cts.reaper.search.tokenLimit advanced server property. The default is 5000.

To configure how often the search runs, and the grace period of the tokens that will be reaped, configure the following properties:

  • org.forgerock.services.cts.reaper.search.pollFrequencyMilliseconds.

    Specifies how often to perform a search for expired tokens. The default is 5000 milliseconds.

    The value of this setting should never be higher than a few seconds, since the goal is to launch the search fair often so that the expired tokens do not pile up.

  • org.forgerock.services.cts.reaper.search.gracePeriodMilliseconds.

    Specifies the grace period used when searching for expired tokens. Any tokens that expired more than the specified duration ago are returned. The default is 300000 milliseconds.

    The grace period should be larger than the value controlled by the org.forgerock.services.cts.reaper.cache.pollFrequencyMilliseconds advanced property. This allows an AM instance sufficient time to delete the token using its cache, rather than search.

    Deleting from the cache is preferred as it avoids expensive searches against the CTS store.

To tune the reaper searches, run load tests in your environment with the default reaper parameters, and then watch if the expired tokens build up. If there is none, then the reaper is sufficiently tuned. If you see token build-up, tune the reaper cache first, since it is the less expensive of the two reaping mechanisms.

It is fine if there is a small amount of expired tokens left on each iteration of the reaper search as long as tokens are cleared up during lower activity periods.

Keeping a large amount of expired tokens in the CTS store will result in increased disk space usage, and in AM not sending expiration notifications in a timely fashion.

Setting the token limit too high could negatively affect the performance of the DS instance. It is better to launch more runs of the reaper search with smaller amounts of records, than fewer runs with larger amounts of records.

Queue size and timeout

One tuning consideration is to manage the CTS queue size and timeout for efficient throughput. AM makes CTS requests from the following components:

  • AM server-side sessions and authentication sessions

  • OAuth 2.0

  • UMA

  • Session denylist (if enabled for client-side sessions)

  • Authentication session allowlist (if enabled for client-side sessions)

  • SAML v2.0 (if enabled for Security Token Service token validation and cancellation)

  • OAuth 2.0 server-side tokens and client-side token denylist

  • Push notification during authentication

  • Cluster-wide notification

Every create, update, and delete requests to CTS are placed into an asynchronous buffer before being handled by an asynchronous processor. This ensures that callers performing write operations can continue without waiting for CTS to complete processing.

Once the queue is full, all new operations are "blocked" before being placed in the queue. Once the queue is frees up, the caller can continue as normal.

CTS is designed to automatically throttle throughput when the buffer fills up with requests. Therefore, if you require a balance between performance versus system memory, AM provides two properties that can be used to tune CTS, queue size and queue timeout.

org.forgerock.services.cts.async.queue.size

Default size: 5000. Determines the amount of request operations that can be buffered before the queue size becomes full, after which the caller will be required to wait for the buffered requests to complete processing. All CRUDQ operations are converted to tasks, which are placed in the queue, ensuring that operations happen in the correct sequence.

org.forgerock.services.cts.async.queue.timeout

Default timeout is 120 seconds. Determines the length of time a caller will wait when the buffer is full. If the timeout expires, the caller receives an error. The timeout property is used in any system configuration where the LDAP server throughput is considerably slower than the AM server, which can result in blocked requests as the backlog increases.

To set the queue size and timeout properties:

  1. In the AM admin UI, go to Configure > Server Defaults > Advanced.

  2. Enter the key name and value.

  3. Click Add.

For additional information on tuning the CTS, see Tune the CTS store LDAP connections.

Virtual attributes

DS supports a number of virtual attributes, which dynamically generate entry values that are not persisted in the token store. By default, the following DS virtual attributes are enabled at install:

  • collective-attribute-subentries

  • entity-tag

  • entry-dn

  • entry-uuid

  • governing-structure-rule

  • has-subordinates

  • is-member-of

  • num-subordinates

  • password-expiration-time

  • password-policy-subentry

  • structural-object-class

  • subschema-subentry

To improve CTS throughput, you can disable your virtual attributes using the dsconfig command, except for the Entity Tag virtual attribute, which must remain enabled else it will lead to an inoperable server.

To disable a virtual attribute, use the dsconfig command on DS to disable, for example, the Collective Attribute Subentries virtual attribute:

./dsconfig set-virtual-attribute-prop \
--name=collective-attribute-subentries-virtual-attribute \
--set enabled: false \
--hostname localhost \
--port 4444 \
--bindDN uid=admin \
--bindPasswordpassword\
--no-prompt

The DS entity-tag-virtual-attribute is required and should never be disabled.

OAuth 2.0 CTS storage scheme

AM 6.5 introduced a new scheme for storing OAuth 2.0 tokens in the CTS store, called the grant-set scheme.

The grant-set scheme groups multiple authorizations for a given OAuth 2.0 client and resource owner pair and stores them in a single CTS OAUTH2_GRANT_SET entry. This implementation reduces the size and quantity of entries stored, as well as the number of calls required to perform OAuth 2.0 operations.

The one-to-one scheme stores the state of multiple authorizations for a given OAuth 2.0 client and resource owner pair across multiple entries, and is more resource intensive. You should upgrade to the grant-set scheme once all the servers on your environment have been upgraded to AM 6.5 or later.

The grant-set scheme is backwards-compatible with existing entries stored in the CTS store. Therefore, any access or refresh token issued before configuring the grant-set scheme is still valid. Existing tokens will be retained in their original form until the refresh token expires or it is actively revoked.

Users will not notice any change in the tokens they receive, and there is no change to the OAuth 2.0 endpoints.

To enable the grant-set scheme:

  1. Go to Configure > Global Services > OAuth2 Provider > Global Attributes.

  2. In the CTS Storage Scheme drop-down, choose Grant-Set Storage Scheme.

  3. Click Save Changes.

New OAuth 2.0 tokens stored in the CTS after the change will use the new scheme automatically.