Access Management 7.3.2

Tune the CTS

There are several tuning considerations for the efficient processing of your CTS token store:

Task Resources

Configure the reaper cache size

The reaper process prunes tokens in the store that have reached their maximum time-to-live.

Reaper cache size

Manage the reaper search size

In addition to the cache, the reaper also uses a search to find expired tokens. You can tune the maximum amount of tokens that the reaper can find and delete in a single invocation.

Reaper search size

Change the OAuth 2.0 Storage Scheme

If the CTS token store was created on a version of AM earlier than 6.5, configure the grant-set scheme to improve the performance of OAuth 2.0-related operations.

OAuth 2.0 CTS storage scheme

Set the poll interval for the logout user cache

In a multi-server deployment, servers poll the CTS at a specified interval to update their logout user token cache with the tokens from other servers.

CTS poll interval

Reaper cache size

The size of the AM reaper cache is controlled by the org.forgerock.services.cts.reaper.cache.size advanced property. The default size is 500000 tokens.

If an AM server is under sustained heavy load, the reaper cache may reach capacity, causing degraded performance due to the additional slower searches of the CTS store. If the reaper cache is full, messages are logged in the Session debug log, such as the following:

The CTS token reaper cache is full. This will result in degraded performance.
You should increase the cache size by setting the advanced server property
'org.forgerock.services.cts.reaper.cache.size' to a number higher than 500000.

If this debug message appears frequently in the debug logs, increase the value of the org.forgerock.services.cts.reaper.cache.size property. To alter the value, in the AM admin UI, go to Configure > Server Defaults > Advanced, and add the property and increased value to the list.

Increasing the size of the reaper cache causes higher memory usage on the AM server. If a cache of the default size of 500000 entries is nearly full, the server memory used could be up to approximately 100 megabytes.

Tune the AM reaper cache size only if the org.forgerock.services.cts.store.reaper.enabled advanced server property is set to true.

Reaper search size

The reaper uses its cache to ensure that expired tokens are removed from the CTS token store, but it also has a built-in mechanism to search for expired tokens that may have not been purged. This could happen, for example, due to an AM instance crash.

When the reaper searches for expired tokens, DS returns a page of records; the reaper deletes these tokens.

To configure the maximum number of records that DS returns, use the org.forgerock.services.cts.reaper.search.tokenLimit advanced server property. The default is 5000.

To configure how often the search runs, and the grace period for the tokens the reaper deletes, configure the following properties:

  • org.forgerock.services.cts.reaper.search.pollFrequencyMilliseconds.

    Specifies how often to perform a search for expired tokens. The default is 5000 milliseconds.

    The value of this setting should never be higher than a few seconds, since the goal is to launch the search fair often so that the expired tokens do not pile up.

  • org.forgerock.services.cts.reaper.search.gracePeriodMilliseconds.

    Specifies the grace period used when searching for expired tokens. Any tokens that expired more than the specified duration ago are returned. The default is 300000 milliseconds.

    Set the grace period larger than the value controlled by the org.forgerock.services.cts.reaper.cache.pollFrequencyMilliseconds advanced property. This allows an AM instance sufficient time to delete the token using its cache, rather than search.

    Deleting from the cache is preferred as it avoids expensive searches against the CTS store.

To tune the reaper searches, run load tests in your environment with the default reaper parameters, and then watch if the expired tokens build up. If there is none, then the reaper is sufficiently tuned. If you see token build-up, tune the reaper cache first, since it is the less expensive of the two reaping mechanisms.

It is fine if there is a small amount of expired tokens left on each iteration of the reaper search as long as tokens are cleared up during lower activity periods.

Keeping a large number of expired tokens in the CTS store results in increased disk space usage, and in AM not sending expiration notifications in a timely fashion.

Setting the token limit too high could negatively affect the performance of the DS instance. It is better to launch more runs of the reaper search with smaller amounts of records, than fewer runs with larger amounts of records.

OAuth 2.0 CTS storage scheme

AM 6.5 introduced a new scheme for storing OAuth 2.0 tokens in the CTS store, called the grant-set scheme.

The grant-set scheme groups multiple authorizations for a given OAuth 2.0 client and resource owner pair and stores them in a single CTS OAUTH2_GRANT_SET entry. This implementation reduces the size and quantity of entries stored, as well as the number of calls required to perform OAuth 2.0 operations.

The one-to-one scheme stores the state of multiple authorizations for a given OAuth 2.0 client and resource owner pair across multiple entries, and is more resource intensive. You should upgrade to the grant-set scheme once all the servers on your environment have been upgraded to AM 6.5 or later.

The grant-set scheme is backwards-compatible with existing entries stored in the CTS store. Therefore, any access or refresh token issued before configuring the grant-set scheme is still valid. The CTS store retains an existing token in its original form until the refresh token expires or the token is actively revoked.

Users do not notice any change in the tokens they receive. There is no change to the OAuth 2.0 endpoints.

To enable the grant-set scheme:

  1. Go to Configure > Global Services > OAuth2 Provider > Global Attributes.

  2. In the CTS Storage Scheme drop-down, choose Grant-Set Storage Scheme.

  3. Click Save Changes.

New OAuth 2.0 tokens stored in the CTS after the change use the new scheme automatically.

CTS poll interval

If client-side session logout is enabled, AM stores the logout user tokens in a local cache. In a multi-server deployment, servers poll the CTS at a specified interval to update their logout user token cache with the tokens from other servers.

The poll interval is specified by the global session property Enable Invalidation of Sessions Based on User Identifier, and is set to 60 seconds by default.

A very long polling interval gives malicious users time to connect to other AM servers within a site and use stolen session cookies. A very short polling interval improves security by invalidating user sessions across servers in a shorter time-frame, but can incur a performance cost.

A value of 0 disables polling of the CTS. Do not disable polling in a multi-server site, as this prevents logoutByUser session invalidation data from being shared between servers.