PingDirectory

Troubleshooting server performance issues

The following are the most common reasons that customers encounter performance issues and some suggestions for diagnosing and addressing these issues.

Slow password storage schemes

Slow password storage schemes are configurable schemes designed to use a lot of CPU and memory to thwart attackers, but they can affect legitimate operations like password validation and authentication.

The following password storage schemes are intentionally expensive:

  • PBKDF2

  • bcrypt

  • scrypt

  • Argon2

  • The MD5, SHA-2-256 and SHA-2-512 variants of the crypt scheme

These schemes are designed to consume a significant amount of CPU, and memory in some cases, to increase the amount of resources an attacker must expend to crack a password if they happen to get access to the password’s encoded representation. This same cost is also incurred for legitimate operations involving the password, including encoding clear-text passwords during account creation and password changes and when validating passwords during authentication. You can configure these schemes to adjust the amount of resources they consume, and you should configure them so that the resource consumption under expected peak load does not exceed the capacity of the topology.

Additionally, if you are initially populating the server using an LDIF import that contains clear-text passwords, using one of these schemes can cause the LDIF import to proceed at a small fraction of the rate that could be achieved with a faster storage scheme, such as one that uses a 256-bit or 512-bit salted SHA-2 digest. In such cases, you might import the data using a faster scheme and then change the configuration to make the desired scheme the new default, and mark the scheme used for import as deprecated. As a result, accounts with passwords encoded using the import scheme are automatically re-encoded with the new scheme the first time that the user successfully authenticates using that password.

Database size versus memory capacity

Best performance is achieved when the contents of the database can be fully cached in memory.

If possible, the amount of memory allocated to PingDirectory server should be large enough so that all of the data can fit in the database cache. Memory sizing estimates appear at the end of the output when you run the import-ldif tool.

If the amount of data that needs to be stored exceeds the available memory capacity of the individual systems on which the server runs, there are a couple options:

  • Break up the data into segments that are small enough to be fully cached on their own, and use PingDirectoryProxy server’s entry balancing functionality to make them appear as a single logical set.

  • Tune the server so that the most frequently accessed information can be served from memory while disk access might be required for other data. In disk-bound deployments, you can use backend configuration options such as cache mode, uncached attribute criteria, and uncached entry criteria to help prioritize what data should remain in memory versus what can be retrieved from disk.

Large number of access control rules

As the number of access control rules increases, so does the potential costs of determining whether a client is allowed to request a given operation and of paring down search result entries based on the data that the client is permitted to access.

The server might need to re-evaluate all access control rules after certain update operations, including modify DN operations, to determine whether these are affected by the change.

In many cases, deployments with an extremely large number of access control rules, especially those with large numbers of branches in which the same structure might be repeated across each of these branches, might be able to leverage parameterized access control instructions (ACIs) to dramatically reduce the number of access control rules that need to be defined and evaluated. In other cases, it is possible to refactor the access control configuration to achieve the same effect but with far fewer rules.

Large static groups

PingDirectory server supports large static groups with hundreds of thousands of members or more, but these can have a significant impact on performance. Consider replacing large static groups with dynamic groups.

It can be expensive in terms of system resources to update the large static group to add or remove members because each update requires rewriting the entire entry. It can also be expensive to decode the entry when retrieving it from the database, even if it is held in the database cache, and to return the list of members to a client in search results.

If possible, consider replacing large static groups with dynamic groups. Dynamic groups automatically determine their membership based on criteria defined in LDAP URLs. A dynamic group consumes little memory and disk space and does not need to be altered as entries are created, updated, and removed. In some cases, it is possible to use virtual static groups in conjunction with dynamic groups to create entries that behave like static groups for read operations. This lets you maintain compatibility with applications that only understand static groups but use dynamic groups behind the scenes to determine membership.

If it is not possible to eliminate large static groups, you can enable the static group entry cache. While this does not reduce the performance impact of updating large static groups, it can make it much more efficient to access those groups for read operations.

Large index ID sets

Indexes store data that make it possible to quickly retrieve matching entries during a search. As the size of an ID set increases, so does the potential resource cost of accessing the ID set.

Each index record maps an index key to a list of the entry IDs for the entries that match that key. If you have an equality index for a given attribute, there is a key for each unique value for that attribute and the values for each key of the IDs of the entries that contain that value. For substring indexes, there can be multiple keys for the same attribute value (one for each unique six-character substring within the value), and the same substring can apply to many different values of the same attribute.

For an attribute index, you can store ID sets in one of two ways that each affect performance differently:

  • In the regular (non-exploded) case, each index key occurs once, and the value of this key is the entire list of IDs for entries that match the key. The ID set for a non-exploded index key can be retrieved quickly because only a single database read is required. However, as the size of the ID set grows, the cost of updating it grows because it is necessary to replace the entire set, which requires larger amounts of disk I/O and can place an increased burden on the database cleaner.

  • In the exploded case, the same key can be stored multiple times (one for each entry that matches that key), with each instance of the key associated with a different entry ID. Updating the ID set for an exploded key is always fast because the writes are small, but the cost of reading the ID set increases with the number of IDs.

You can also use composite indexes. These offer many advantages over attribute indexes and when they can be used. Some of these advantages, such as the ability to configure a base DN or combinations of attributes, do not have any effect on performance with regard to large ID sets. They use a hybrid of the exploded and non-exploded approaches to maintaining the ID set, such that a large set can be split into multiple pieces, but each piece can have up to 5000 IDs rather than just one. This means that retrieving a large ID set from a composite index can be thousands of times cheaper than retrieving the same ID set from an exploded attribute index. Updating a large ID set in a composite index should be cheaper in terms of systems resources than updating the same ID set in a non-exploded attribute index because the write is much smaller.

In environments with performance problems related to very large index ID sets, you might consider the following options as a way to help improve performance:

  • Consider reducing the index entry limit for that index. The index entry limit specifies the maximum number of IDs that an ID set can have for an index key.

    • If a key matches more entries than this limit, the server stops maintaining the index for that key, and attempts to access it behave as if it were unindexed, while the index continues to be maintained for keys matching smaller numbers of entries. If you don’t have searches that depend only on those keys, then this is an excellent way of eliminating the cost of maintaining large ID sets. It isn’t logical to set the index entry limit to a value that is a large percentage of the total number of entries in the server. In such cases, there might not be a significant performance difference between indexed and unindexed search performance, but there would be no need to maintain the associated large ID sets.

  • If there are cases in which you need a large index entry limit, then consider increasing the limit only for that index, rather than increasing the default limit for all indexes in the backend.

    • The index-entry-limit property in the backend applies to all indexes that don’t specify their own limit, but each index also offers an index-entry-limit property that, if set, overrides the index entry limit configured for the backend. As such, if you need a higher index entry limit for a particular index, set a higher limit just for that one index instead of raising the default limit for all indexes in the backend.

  • For attribute indexes with keys matching a large number of entries, consider converting it to a composite index when possible.

    • Composite indexes can completely replace equality and ordering attribute indexes, and they can support “starts with” substring searches, regardless of whether they have “contains” or “ends with” components. Composite indexes can’t currently replace approximate match indexes or substring searches that don’t have a “starts with” component.

  • Consider eliminating any unnecessary substring indexes.

    • As previously noted, substring indexes are more likely to have large ID sets than equality, ordering, or approximate match indexes because substring keys are generally smaller and any given substring key can apply to multiple attribute values. It’s also not commonly understood that equality attribute indexes and equality composite indexes are used for substring searches with a “starts with” component. As such, a substring index is not used for substring searches with a “starts with” unless there isn’t an equality index for that attribute.

    • If a substring index is defined for an attribute that isn’t targeted by substring searches, or that is only targeted by substring searches that contain a “starts with” component (regardless of whether it also includes “contains” or “ends with” components), then that substring index is not necessary and can be removed. You can use access logs to determine the types of searches that clients are performing, and the summarize-access-log and search-logs tools might help with that.

  • If a substring index is needed for a given attribute, then consider increasing the substring length for that index.

    • By default, the server creates a separate substring key for each unique six-character substring in an attribute value, and there might be cases in which the same six-character substring appears in several different values. If that occurs and causes substring keys to have large ID sets, then increasing the substring length for that index reduces the number of values that might share the any given key and can reduce the number of IDs associated with that key.

  • As a last resort, consider tuning the exploded index threshold for an index (the number of entries that an ID set needs to have for a given key before it will transition from a non-exploded set to an exploded one) based on the expected usage for that index.

    • If search performance is more important than update performance for an attribute with large ID sets, then raising the exploded index threshold helps keep the ID set stored as a monolithic block of IDs. On the other hand, if update performance is more important than search performance, then lowering the exploded index threshold might help.

Missing indexes

Maintain indexes to determine if an index is missing.

Maintaining unneeded indexes can have a detrimental effect on performance, but it can also be detrimental if a needed index is missing.

The best way to identify expensive searches processed in the server is to examine the access logs for search operations with a high etime (elapsed processing time) value. After you identify these search operations, you can filter out any of these operations that do not need to be fast. For example, you might have applications that generate reports by performing inefficient searches, such as searches to retrieve all entries, and it is usually more acceptable for those searches to be slow.

For any remaining searches that are slow or that should be faster, the best way to understand why the search is expensive is to issue a search with the same base DN, scope, and filter, but request only the debugsearchindex attribute.

debugsearchindex is a special attribute that makes the server return debug information about the index processing that is being performed in the course of evaluating the search, and how long it took to complete each step of the evaluation.

From this output, you can see which indexes were used and which could not be used because there was either no applicable index or the index entry limit had been exceeded for the target key. You can see expensive accesses to exploded indexes and identify indexes you want to add, indexes that can benefit from being converted to composite indexes, or indexes where you might need to increase the index entry limit. Alternatively, you can determine a different way to perform the search so that it does not depend on components that are unindexed or that match a large number of entries.