Upgrade Considerations

Important considerations for upgrading to this version of the Directory Server

  • The Delegated Admin web app now supports creation of new users. Installations created using older versions of the install script require a command like the following to be run after upgrade. The 'sn' attribute is a required attribute for inetOrgPerson entries.

    dsconfig create-delegated-admin-attribute --type-name users --attribute-type sn --set "display-name:Last Name"

    To enable user creation, one of the new configuration properties org-entry-dn or org-search-filter must be set on the Delegated Admin resource type.

What's New

These are new features for this release of the Directory Server

  • Introduced a Directory REST API to create, read, update and delete (CRUD) any object in the directory using JSON over HTTP. Compared to the SCIM-based Identity Access API (introduced in 4.0), the Directory REST API offers more capability without the configuration overhead and SCIM protocol limitations. See the Directory REST API documentation included in the “docs” folder for more information.

  • Overhauled the way in which the directory evaluates and processes search expressions. The new search planner leverages index statistics in order to process search expressions in the most effective order. The search planner now considers the overall size of each index, specific values that have too many matches, and the relative distribution of values.

  • Improved composite indexes. Composite indexes are used to improve performance for searches where attributes are often searched together, like a customer tenant identifier alongside a person’s name. Whereas previously you could only index a combination of the DN of ancestor objects with a single attribute of descendant objects, now you can define a composite index using a pattern that combines equality expressions and at most one substring expression. For example, you can create a composite index covering the entire expression (&(tenantId=?)(cn= \ *? \ *)).

  • Improved integration with some third-party software. Now the changelog backend (which makes available all recent changes to directory entries) supports paging over LDAP using the Simple Paged Results Control (RFC 2696). This enables third-party data integration software to more efficiently obtain a list of recent changes over LDAP. Previously the Simple Paged Results Control only worked for LDAP searches against user data stored in backends like userRoot.

  • Improved the scheduled tasks feature designed to help automate maintenance tasks on the directory server. Now you can schedule the execution of whitelisted commands on the directory server host, create routine LDIF exports for multiple backends, make the server administratively enter or leave “lockdown” mode, and clean up old files from disk, like historical log files and LDIF exports.

  • Improved user and group management in the delegated user administration web app (packaged separately.) Administrators can configure the presentation order and groupings of profile attributes, improving delegated admin usability for large user profiles. Also, delegated admins can now create new users, and add and remove entire sub-groups of users to and from groups.

  • Added support for Oracle Java JDK 11 and OpenJDK 11. Added support for RedHat 7.5, CentOS 7.5, and Ubuntu 18.04 LTS. When running on JDK 11, we now configure G1GC as the default garbage collection algorithm. This eliminates long garbage collection pauses in most environments.

  • Improved event tracing across coordinating HTTP and directory servers. For example, admins can now trace requests to HTTP-based services, like the Directory REST API, through the log files to the LDAP access log of the Directory Server. All HTTP-based services (e.g. SCIM, Consent API) can be configured to accept and/or generate HTTP request headers with correlation identifiers. These identifiers are logged in trace logs, HTTP access logs, and LDAP access logs.

  • Added a new command line tool, extract-data-recovery-log-changes, to assist in replaying or reverting data changes, for example, to aid in disaster recovery or to back out changes made in error. The new tool extracts relevant changes from the server's audit logs and produces an LDIF file that can be imported by ldapmodify. Admins can select or filter changes based on numerous criteria.

Known Issues/Workarounds

The following are known issues in the current version of the Directory Server

  • There are known issues when running the server with Java 11.0.0. These are addressed in Java 11.0.1. In general, when using Java 11, we recommend using the latest available release.

  • The Identity Access API has been deprecated and will not be supported in the next major release.

  • On Microsoft Windows systems, JVM arguments for verbose GC logging do not work as expected. So these arguments are not added to any of the server or client tools.

Resolved Issues

The following issues have been resolved with this release of the Directory Server:

Ticket ID Description

Fixed an issue causing unexpected crashes in dsconfig when backing out of certain screens.

DS-1132

Updated the server to allow delaying the response to failed bind operations by a specified length of time. While the response is delayed, no other operations will be allowed on the connection. This can be used instead of, or in addition to, account lockout as a means of limiting the rate at which an attacker may try to guess user passwords.

DS-5964

Fixed an issue where an entry could be added to the server with invalid privileges.

DS-6930

To facilitate testing in multiple GC (garbage collection) environments, GC JVM options having been moved to separate Java properties in the java.properties file. The new ".gc-type" suffix will select the GC type to use, and the new ".gc-<GC type>-args" suffix will have the JVM options for that GC type.

DS-14799

Updated the server to support sending persistent search results asynchronously, which protects against a blocked persistent search client from interfering with write operation processing.

DS-17771

Fixed an issue preventing third-party tasks from loading correctly.

DS-18261,DS-35906

Added an extract-data-recovery-log-changes tool that can be used to examine a server audit log file (preferably the data recovery log file, which is preconfigured for optimal compatibility) to extract changes matching a given set of criteria, including change time, operation type, requester DN, client address, change content, and more. The extracted changes can be formatted as they were originally requested so that they can be replayed, or they can be inverted so that they can be backed out.

DS-35516

Fixed an issue where the server would not prevent an invalid entry with more than one structural object class from being added, if any of those classes was a groupOfURLs.

DS-35553

Fixed bug where named pipe can't be used as log file.

DS-35567

Updated the server to expose information about the duration of lengthy phases of server startup. The longest phases are logged to the error log and more detail is provided in the "cn=Startup Phase Times,cn=monitor" monitor entry. Starting the server with the --verbose option will show fine-grained timing information for all phases of server startup.

DS-35652,DS-36559

Added support for a new file retention task that can identify files in an indicated directory that match a given pattern and remove any matching files that fall outside of the specified retention criteria. You can specify the minimum number of files that should be retained, the minimum age of files that should be retained, the minimum aggregate size of files that should be retained, or any combination thereof. The files that match the pattern will be sorted by timestamp so that if any files are to be removed, the most recent files will be retained and the oldest files will be deleted.

The file retention task can be scheduled as a standalone task or as a recurring task. Two instances of the file retention recurring task have been defined in the default configuration: one that can clean up old expensive operation dump files, and another that can clean up old work queue backlog thread dump files. In each case, the recurring task is configured to keep at least the 100 most recent files, and no files less than 30 days old will be removed. While these recurring tasks are defined in the out-of-the-box configuration, they are not part of any recurring task chain and therefore will not actually be invoked unless they are configured as part of a chain.

The Directory Server and Directory Proxy Server now include recurring tasks in the out-of-the-box configuration that can clean up old expensive operation dump log files or work queue backlog thread dump log files if too many of them have collected in the server logs directory. For each type of file, if there are more than 100 of them in the server logs directory, then any of the remaining files that are more than 30 days old are candidates for removal. A recurring task chain will perform this cleanup every day at 12:05 a.m. in the JVM's default time zone.

DS-35865

Multiple instances of the SCIM HTTP Servlet Extension may now be created, allowing for multiple SCIM 1.1 service configurations per server instance. For more information, please refer to the "Managing the SCIM Servlet Extension" chapter of the Administration Guide.

DS-35873

Added support for an exec task that can invoke commands on the server. There are several safeguards in place to prevent unauthorized users from invoking arbitrary commands on the server system, including a new exec-task privilege and a whitelist file that must be updated to include the absolute paths of the allowed commands. A new schedule-exec-task tool helps create an exec task from the command line, and the LDAP SDK has also been updated to allow interacting with exec tasks programmatically.

DS-35873

Added support for recurring exec tasks.

DS-35909

HTTP Connection Handlers will now raise an alarm during initialization if a context path conflict is detected.

DS-36023,DS-37686

A new --topologyFilePath argument has been added to remove-defunct-server, making it possible to remove a defunct server cleanly from the topology using one of the servers in the provided topology file. The topology file may be obtained by running the manage-topology export command.

DS-36030

Fixed an issue that could prevent the server from imposing the correct size and time limits for search requests with an alternate authorization identity.

DS-36132

Fixed an issue in the backup tool where --signHash could be used without --hash.

DS-36209

A header containing a correlation ID is now added to outgoing HTTP servlet responses, allowing HTTP responses to be correlated with log messages across server instances. The name of the correlation ID response header defaults to "Correlation-Id" but may be changed by setting the HTTP Connection Handler's correlation-id-response-header property. By default, the server will generate a globally unique correlation ID automatically, but the correlation-id-request-header configuration property may be used to optionally specify one or more request headers that provide an existing correlation ID value from the requesting client. The correlation ID header can be disabled on a per-HTTP Connection Handler basis using the use-correlation-id-header configuration property.

For Server SDK extensions that have access to the current HttpServletRequest, the correlation ID can be retrieved as a String via the HttpServletRequest's "com.pingidentity.pingdata.correlation_id" attribute. For example: <code> \ (String) request.getAttribute \ ("com.pingidentity.pingdata. \ correlation_id"); \ </code>

DS-36433

Added a Mock Access Token Validator, which accepts access tokens without validating the authenticity of the tokens using a trusted authorization server or signing certificate. When enabled, a Mock Access Token Validator accepts bearer tokens in the form of a plain text JSON object containing an arbitrary set of claims. Mock Access Token Validators are intended for test or demonstration use only and should never be enabled in production deployments or used to access sensitive data.

DS-36453

Updated the server to enable automatic LDIF exports by default for new installations. Every day at 1:05 a.m. (in the JVM's default time zone, which is generally the time zone configured for the underlying system), the server will export the contents of each non-administrative backend to a file in the "ldif" directory immediately below the server root. The LDIF exports will always be compressed, and they will be encrypted if the global configuration is set to encrypt LDIF exports by default (which will be enabled if encryption is configured during setup). The LDIF exports will be rate limited to ten megabytes per second to minimize the impact on server performance, and exports will be retained for seven days.

Daily LDIF exports will only be enabled by default for new installations. The recurring task chain will be created in instances that are updated to this release, but that chain will not be enabled.

DS-36483

Fixed a defect where the compression-mechanism and compression-parameter properties were not hidden in ReplicationServerConfiguration.xml.

DS-36510

Added support for a delay task, which can be used on its own or as a recurring task. It is primarily intended to be used as a spacer between other tasks, and can sleep for a specified period of time, wait for the server to be idle (that is, there are no outstanding operations and all worker threads are idle), or wait for sets of search criteria to match at least one entry (for example, until a monitor entry indicates that the server is in a desired state).

DS-36511

Added a plugin that supports encrypting the values of operational attributes intended to hold sensitive information, including TOTP shared secrets, delivered one-time passwords, password reset tokens, and single-use tokens.

DS-36519

Bearer token authentication for the Consent API may now be enabled or disabled using the bearer-token-auth-enabled property of the Consent HTTP Servlet Extension.

DS-36585

Updated the client connection policy configuration to add a maximum-concurrent-operations-per-connection-exceeded-behavior property that specifies the behavior that the server should exhibit if a client tries to exceed the limit set by the maximum-concurrent-operations-per-connection property. Previously, any requests in excess of the maximum-concurrent-operations-per-connection limit would have been rejected with a busy result. The server now offers additional choices for the result code to use when rejecting requests (including admin limit exceeded, constraint violation, unavailable, unwilling to perform, or other), and the server can also be configured to close the connection and abandon all outstanding operations on that connection.

DS-36592

Enhanced rebuild-index performance, especially in environments with encrypted data. This prevents work queue backlogs that could occur when rebuild-index runs with the server online.

DS-36694

Replication now sends heartbeat and monitoring information less frequently to reduce the high network overhead that had been observed in topologies with more than fifty servers. The interval between monitoring data updates is now configurable through the remote-monitor-update-interval property on the Replication Server configuration object.

DS-36729

Added an attributes-modifiable-with-ignore-no-user-modification-request-control global configuration property that will a select set of operational attributes declared with the NO-USER-MODIFICATION constraint to be updated in a modify request that includes the ignore NO-USER-MODIFICATION request control. At present, this is only supported for the creatorsName, createTimestamp, modifiersName, and modifyTimestamp attributes.

DS-36801

Fixed an issue with the Dictionary Password Validator where configuring case-sensitive-validation=false would only work if the input file included the lower-case version of all passwords. The server now automatically converts the passwords to lower-case in memory when configured with case-sensitive-validation=false.

DS-36834

All tools will now enforce a minimum heap size requirement. Overriding the heap size for the system, using the --maxHeapSize argument of the dsjavaproperties tool, is only effective if the provided value is greater than the minimum required heap size for the tool.

DS-36910

Fixed an issue in which the server could return an incorrect result code for add and modify DN requests that included a malformed DN.

DS-36988

The SCIM v1 servlet extension is no longer enabled by default for new installations. Existing installations will be unaffected on an upgrade. Customers are encouraged to use the new "Directory REST API" for REST access from now on.

DS-37018

Added a Constructed Virtual Attribute that can be used to dynamically construct values for a virtual attribute using a combination of fixed text and values of other attributes from the entry. As with most virtual attributes, searching on the attribute is not indexed.

DS-37037

Updated the recurring LDIF export task to support exporting the contents of multiple backends.

DS-37053

Fixed an issue where manage-extension could install files outside of the installation directory.

DS-37066

Added recurring task support for placing the server in lockdown mode and taking the server out of lockdown mode. While in lockdown mode, the server reports itself as unavailable to the Directory Proxy Server and only accepts requests from a restricted set of clients.

DS-37131

Fixed an issue in which the HTTP Servlet Config Monitor could cause an exception in an HTTP Servlet Extension when attempting to determine its context paths. This caused the status tool and the Administrative Console to potentially omit the HTTP Servlet Extension from the list of active HTTP extensions.

DS-37134

The "dsreplication" command now has a "--topologyLockWaitSeconds" option that controls how long subcommands will wait for the topology lock. This is helpful when running multiple subcommands that each require the topology lock. Without this option, subcommands that require the topology lock will fail if it is not available.

DS-37176

Added three new matching rules that can be used to customize equality matching for attributes with a JSON object syntax. Previously, it was only possible to perform equality matching with case-sensitive field names and case-insensitive string values. The new matching rules add support for other permutations of case sensitivity for field names and string values.

DS-37181

Updated the out-of-the-box configuration to clean up old lock conflict details log files if too many of them have accumulated. If there are more than 100 of them in the server logs directory, then any of the remaining files that are more than 30 days old are candidates for removal. The cleanup will occur every day at 12:05 a.m. in the JVM's default time zone.

DS-37251

The exploded-index-entry-threshold configuration property on Local DB Backend and Local DB Indexes has been hidden because it is poorly understood and often misconfigured. The effective value of this property is now 50000 since it allows good search performance and high bursts of write throughput.

DS-37262

Updated num-worker-threads and *-worker-thread-percent-busy attributes to exclude admin queue threads from the worker thread count. These attributes will now better reflect the actual number of worker threads available.

DS-37313

Addressed an issue where an InvalidKeyException could occasionally be reported by import-ldif. The error message for this problem resembles, "An unexpected error occurred during merge processing for index 'dc_example_dc_com_sn.equality': InvalidKeyException: The provided passphrase is invalid."

DS-37352

Updated the audit log to provide the option to include a number of additional fields, including:

* The server product name.

* The server instance name.

* The OIDs of any controls include in the request.

* Details of any intermediate client or operation purpose request controls include in the request.

* Whether the operation was replicated.

* Whether the operation was an internal operation.

* Whether the operation was processed by an administrative session worker thread.

In addition, operation-specific log messages can include the following additional fields:

* For add operations, the log can now indicate whether the operation was an undelete.

* For delete operations, the log can now indicate whether the operation was a soft delete, whether the operation was a delete of a soft-deleted entry, and whether the operation was the base or a subordinate entry of a subtree delete. Further, virtual attributes are separated from real attributes in the record of the deleted entry.

* For modify operations, the log can now indicate whether the entry was a modify of a soft-deleted entry.

DS-37356

The replication server now has a configuration property "replication-purge-minimum-retain-count," which is similar to the existing configuration property "replication-purge-delay" except that a minimum number of changes are enforced instead of a maximum age. The "replication-purge-minimum-retain-count" property may be helpful in those cases where a replication server will experience long delays, (exceeding the "replication-purge-delay") of infrequent traffic while not connected to the other replication servers.

DS-37389

Added support for the simple paged results control to the changelog backend.

DS-37416

Fixed an issue in support for the get effective rights request control that could cause the server to incorrectly report that an anonymous user could have read access to an entry if there are any ACIs that make use of the "ldap:///all" bind rule. The issue affected only get effective rights processing and did not actually expose any server data to unauthorized users.

DS-37420

Addressed an issue that prevented adding or removing users with Identify References virtual attributes if those attributes were included in the changelog backend.

DS-37480

Replication now waits more reliably on startup for missing changes by waiting until there is replica information for all operational replication servers. Also, the server will now wait if a remote server has more recent changes for a local replica.

DS-37492

Added a time limit retention policy to support removing log files older than a specified age.

DS-37515

Updated the server to include a data recovery log in the default configuration. This is an audit log with a configuration optimized for enabling replay or reversion of changes should the need arise. The logger will be defined in the configuration for all new installations and updates of existing installations, but it will only be enabled by default for new installations. The log will always be compressed, and it will be encrypted if data encryption is enabled in the server.

DS-37597

Fixed a bug where the totalResults value for SCIM requests using page parameters would be incorrect if the used LDAPSearch element had more than one baseDN defined in the scim-resources XML file.

DS-37660

Updated the Work Queue to increase the number of internal queues when num-queues is configured with the default of 0 (i.e., the server automatically determines the value). An internal queue is now created for every two worker threads rather than eight. This can reduce thread contention and increase throughput when under extreme load.

DS-37713

Updated the Consent API to infer actor and/or subject values where possible:

* The /consent/v1/consents endpoint no longer requires query parameters to be provided. If neither the subject parameter nor the actor parameter is specified, then the subject will be assumed to be the authenticated identity's user ID.

* When creating or updating consent records, a client is no longer required to provide subject or actor values; these values will be inferred from the authenticated identity. For unprivileged clients, these values are always set automatically and are ignored when set explicitly. Privileged clients may explicitly specify subject and/or actor values, however.

* The Consent Service consent-record-identity-mapper configuration property is no longer needed to support unprivileged clients and is now only needed to support privileged clients that manage consent records on behalf of other users.

DS-37723

Added an optional "titleText" field to Consent API consent definition localization objects. This field may be used to store a localized title or summary for a consent request. A corresponding "titleText" field has also been added to consent record objects.

DS-37778

Updated the Purge Expired Data Plugin to allow the subtree beneath an expired entry to be deleted as well. This is configured by setting the purge-behavior property to subtree-delete-entries.

DS-37823

Fixed an issue where Sensitive Attributes defined on the Global Configuration were incorrectly evaluated for replicated operations. This could lead to failed replicated operations and servers becoming out-of-sync.

DS-37898

Replicated password changes no longer cause password-reset notifications on receiving servers.