PingFederate Server

Configuring runtime thread bulkheads

Set runtime thread bulkheads that limit the percentage of threads that can be waiting on any particular external service.

About this task

Slowdowns or outages in external services such as datastores, protocol endpoints, and other connections, can cause slowdowns in PingFederate while runtime threads wait for responses from these external services. If enough threads are waiting for responses, it can cause thread pool exhaustion, and potentially impact PingFederate’s ability to interact with services that are otherwise not affected.

You can configure runtime thread bulkheads to limit the maximum percentage of the thread pool that can be consumed by a service. For example, if you have a maximum number of 200 runtime threads, and you set your bulkhead to 0.8, single sign-on (SSO) requests that make use of a degrading Lightweight Directory Access Protocol (LDAP) datastore will allow a maximum of 160 threads to wait for that datastore to respond. Further requests to that service are rejected, and the remaining 40 threads can continue to service other requests.

The default bulkhead limit of 0.8 frees up 20% of the total threads for other PingFederate runtime processes. Setting the bulkhead limit too low can cause PingFederate to unnecessarily reject requests. The default bulkhead limit should work for most configurations.

You can also configure PingFederate to reduce the maximum number of threads in use when the default bulkhead limit is reached. When BackOffEnabled is set to true, the maximum number of threads allowed for the degrading external service is lowered to the BackOffThreadPoolUsage limit for a period of time defined by BackOffDurationSeconds. This can further reduce the impact to PingFederate in cases where the external service degradation is not expected to be resolved quickly.

For example, with a BackOffThreadPoolUsage of 0.4, and a BackOffDurationSeconds of 600, the bulkhead limit is lowered to 80 threads for the duration of 600 seconds. During this period, new requests are rejected until thread usage drops below 80 threads. After 600 seconds, the bulkhead limit is increased to 160 threads.

If the external service continue to block 160 threads after the BackOff period, BackOff state is reentered.

Depending on your system average thread pool usage, and its tolerance to external service degradation, you might want to adjust your bulkhead configuration differently so that requests fail sooner when degradation is detected.

For example, if it’s expected that an LDAP datastore will only use a small percentage of the thread pool, you can lower the bulkhead limit to 0.2 so that only 40 threads can be wait for LDAP datastore response while the remaining 160 threads an service other requests.

Bulkheads apply to the following external services:

  • JDBC, LDAP, REST, DynamoDB, and custom datastores. Learn more in Datastore Integration on System requirements.

  • PingOne connections.

  • OAuth and SAML protocol calls.

  • Kerberos KDC requests.

  • External HTTP calls.

Bulkheads currently do not apply to the following:

  • CRL and OSCP lookups

  • HSM integrations

  • Log4J appenders

  • Plugins

You can configure PingFederate to notify administrators when a bulkhead reaches a warning limit defined by ThreadPoolUsageWarningThreshold, as well as when it reaches the maximum limit.

Steps

  1. Go to System → Monitoring & Notifications → Runtime Notifications.

  2. Select the Notification for Bulkhead Alert Events checkbox.

    Learn more about bulkhead monitoring in Configuring runtime notifications.

  3. Click Save.

  4. To configure the settings for runtime threads bulkheads, open the <pf_install>/pingfederate/server/default/data/config-store/com.pingidentity.common.util.resiliency.BulkheadManagerImpl.xml file.

  5. Configure the following parameters:

    Parameter Description

    Enabled

    Whether runtime thread bulkheads are enabled.

    Supported values are true or false.

    The default value is true on new PingFederate installations and false on upgrade.

    MaxThreadPoolUsage

    The percentage of the runtime thread pool that can be consumed by an external resource before bulkhead capacity is reached .

    Requests beyond this percentage will be rejected with a BulkheadException.

    Supported values are a decimal of 1. For example, 0.8 is 80%.

    The default value is 0.8.

    ThreadPoolUsageWarningThreshold

    The percentage of runtime thread pool usage at which PingFederate generates a warning notification.

    Supported values are a decimal of 1. For example, 0.7 is 70%.

    The default value is 0.7.

    Set this value lower than the value for MaxThreadPoolUsage to provide early detection of external system degradation.

    NotificationQuietPeriodMinutes

    The minimum amount of time, in minutes, before another bulkhead notification will be generated.

    The default value is 15.

    BackoffEnabled

    Whether BackOff is enabled.

    When a bulkhead is triggered, PingFederate will reduce the bulkhead threshold to a lower limit defined by BackOffThreadPoolUsage for the duration specified in BackOffDurationSeconds.

    Supported values are true or false.

    The default value is false.

    BackOffThreadPoolUsage

    The percentage of runtime threads that can be used by a bulkhead when in BackOff state.

    Supported values are a decimal of 1. For example, 0.4 is 40%.

    The default value is 0.4.

    BackOffDurationSeconds

    The amount of time, in seconds, that the BackOff state will be active.

    The default value is 30.

  6. After making your changes, save the com.pingidentity.common.util.resiliency.BulkheadManagerImpl.xml file.

  7. Depending on the clustering mode of your deployment, either:

    Choose from:

    • In a standalone environment, restart PingFederate.

    • In a clustered environment, replicate the PingFederate configuration.