Upgrade Considerations

Important considerations for upgrading to this version of the Directory Server

  • This release introduces significant changes to the way servers in a topology are configured with information about each other. Once a server has been upgraded from a pre-7.0 version to 7.0 or later, reverting to the previous version is not supported. Before beginning the upgrade process, make sure you have read and understood the Administration Guide's chapter "Upgrading the Server".

  • SCIM 2 error responses, including Config API error responses, now represent the "status" field as a JSON string rather than as a number. Clients written to expect the earlier version format will need to be updated. In particular, clients written using the SCIM 2 SDK for Java should upgrade to version 2.2.0 or higher.

  • Indexes are now automatically encrypted when data encryption is enabled on the server. For deployments in which data encryption is already enabled, it is recommended that you export the data to LDIF and re-import it to ensure that its indexes are also encrypted.

  • The Administrative Console now uses server information found in the topology registry to populate its server selection control. If the Console is used to manage a legacy server that does not use the topology registry, then the server selection control will not be populated. To manage a different server, the administrator will need to log out of the Console and provide the other server's connection details from the login page.

What's New

These are new features for this release of the Directory Server

  • Added support for encrypting indexes. Enabling encryption does have a small performance impact. The exact impact varies based on the data set and load. In our testing,
    • The in memory and on disk footprint is increased by about 30%.
    • Search rate is decreased by about 20% (but 50K searches/sec is still achievable in a fully-cached environment)
    • Peak add / delete rate is impacted the most, because more indexes are updated.
  • Added a new delegated user administration web application (packaged separately). This web UI enables the server administrator to delegate profile management and some privileged operations to others within the enterprise.

  • Made several security improvements for backups, LDIF exports, and log files. Encryption can be done using a specified passphrase, making it easier to decrypt and use files outside of the server cluster, such as when importing them or reviewing them on a different server. Also, servers can be configured to encrypt backups and exports by default, so that administrators do not inadvertently create an unencrypted artifact.

  • Added the capability to automatically purge data after the data has expired. This feature can be applied to delete directory entries, like new accounts that have not fully completed the signup process, or PingFederate persistent grants that have expired. This feature can also be applied to delete individual JSON attributes of a directory entry. Purging expired data is enabled through the Purge Expired Data plugin.

  • Added new schema and REST API that can be used to build a system to collect, enforce, and manage fine-grained data authorizations or privacy preferences for users stored within the directory.

  • Added a new feature for running recurring maintenance tasks, like backup and LDIF export, without the dependency on Linux cron or Windows schedulers.

  • Simplified management tasks related to configuring servers in a large cluster topology or in an automated deployment. Most notably, servers can now be added to a cluster while other servers are offline.

  • Added a new syntax of Access Control Instructions (ACI) that can dramatically reduce the number of nearly identical ACI used in multi-tenant or multi-organization directories. The new format is parameterized by attributes of the binding user’s DN. For example, the server can extract the “HR" value from a binding user such as uid=Fred,o=HR,dc=data, and use that value to check membership within the related privileged group cn=Admins,o=HR,dc=data. A single parameterized ACI can be written to support groups HR, Finance, IT, Development, Marketing, and so on.

  • Added management features for SSL/TLS certificates. The default certificates used in inter-server replication can be replaced; validation of client certificates for HTTPS-based services like the SCIM REST API can be configured; and you can reload from the trust store for HTTPS client certificates without restarting the server or the HTTP-based services.

  • Added support for these operating system versions: Ubuntu LTS 16.04, CentOS 7.4, RedHat Linux 7.4, SUSE Enterprise 12 SP3

Resolved Issues

The following issues have been resolved with this release of the Directory Server:

Ticket ID Description
DS-426

Added the ability to generate administrative alert notifications when a task starts running, when it completes successfully, or when it fails to complete successfully. Also added the ability to send an email message to a specified set of users when a task starts running or completes successfully, which complements the existing ability to send an email message when a task fails to complete successfully or when it completes with any state, regardless of success or failure.

DS-426

Added support for recurring tasks, which can be used to automatically invoke certain kinds of administrative tasks based on a specified schedule.

At present, only certain kinds of tasks can be scheduled as recurring tasks. This includes both backups and LDIF exports, each of which provides retention support to limit the amount of disk space that the backups and LDIF files consume. It also includes support for any kind of task in which each instance of the task should use exactly the same values for all of the task-specific attributes. The Server SDK also provides an API for creating custom third-party recurring task implementations.

DS-4406

Implemented invocation logging for several server tools, which will write to logs/tools/tool-invocation.log by default upon startup and shutdown. Some of the information recorded by log entries include the tool's start and completion times, the command-line arguments used to initialize them, and the name of the system account used to launch the tool. To modify this behavior, edit the config/tool-invocation-logging.properties file.

DS-4570,DS-14281,DS-14282,DS-14283,DS-14284,DS-17197,DS-17366

The admin backend and the tool used to manage it, dsframework, have been replaced by the topology registry and dsconfig, respectively. The topology registry is automatically mirrored across all servers in the topology, so administrative information is kept in-sync on all servers at all times.

DS-5930

Updated ACIs to add support for parameterized DNs. Using the new syntax, attribute values will be extracted from matching resource DNs and substituted into parameterized target DNs and bind rule group DNs. Parameterized ACIs can be used in a multi-tenant environment, where each tenant has an administrators group. In the past, an ACI needed to be duplicated for each tenant. Now, one parameterized ACI can be used. For example, the parameterized ACI

(target="ldap:///o=($1),dc=example,dc=com")(version 3.0; acl "Subtree Admin Group members may search for and read entries in their subtree."; allow (search, read) groupdn="ldap:///cn=Subtree Admin Group,ou=groups,o=($1),dc=example,dc=com"

allows members of a group with DN "cn=Subtree Admin Group,ou=groups,o=Customers, dc=example,dc=com" to search for and read entries in the "o=Customers, dc=example,dc=com" subtree, allows members of a group with DN "cn=Subtree Admin Group,ou=groups,o=Partners,dc=example,dc=com" to search for and read entries in the "o=Partners, dc=example,dc=com" subtree, and so on for any substitution value for the "($1)" parameter variable.

DS-6850

Added a new Purge Expired Data Plugin that can be used to delete entries or JSON attributes that have expired. The plugin has several configuration properties that include controlling how expired data is identified and how it is purged.

DS-6970

Added support for encrypted logging, using a key generated from an encryption settings definition. Encrypted log files may be decrypted with the encrypt-file tool.

DS-12087

Enhanced the implementation of filters having to do with "changeNumber" in the "changelog" backend so that it now correctly interprets filters that include all changes.

DS-12157,DS-35896

Made a number of improvements to backend backup and restore, and to LDIF export and import:

* Added the ability to encrypt backups and LDIF exports with a key generated from a user-supplied passphrase or with a key generated from an encryption settings definition. Previously, encrypted backups and LDIF exports only used a secret key that was known only to servers within the replication topology. The new options make it easier to restore encrypted backups and import encrypted LDIF files in servers outside of the replication topology. The encrypt-file utility can be used to decrypt encrypted backups and LDIF exports, regardless of how the encryption key was obtained.

* Added the ability to limit the rate at which backups and LDIF exports will be written to disk, which can help avoid performance problems that result from these operations saturating the disk subsystem.

* Added new global configuration properties for automatically encrypting backups and LDIF exports by default, which will be set to true if data encryption is enabled during setup.

* Added new global configuration properties that can specify which encryption settings definitions will be used to obtain the encryption keys for automatically encrypted backups and LDIF exports. If not specified, then the server will use its preferred encryption settings definition, or an internal topology key if no encryption settings definitions are available.

* Added a new configuration property for automatically compressing encrypted LDIF exports.

* Updated the backup tool to add new --promptForEncryptionPassphrase, --encryptionPassphraseFile, and --encryptionSettingsDefinitionID arguments that can be used to specify which key to use for encrypting the backup. Added a new --doNotEncrypt argument that can be used to force a backup to be unencrypted even if automatic encryption is enabled. Added a new --maxMegabytesPerSecond argument that can be used to impose a limit on the rate at which the backup may be written to disk.

* Updated the restore tool to add new --promptForEncryptionPassphrase and --encryptionPassphraseFile arguments that can be used to provide a user-supplied passphrase for use in accessing the contents of an encrypted backup. For backups encrypted with an encryption settings definition or an internal topology key, the server will automatically be able to determine the correct key.

* Updated the export-ldif tool to add new --promptForEncryptionPassphrase, --encryptionPassphraseFile, and --encryptionSettingsDefinitionID arguments that can be used to specify which key to use for encrypting the export. Added a new --doNotEncrypt argument that can be used to force an LDIF export to be unencrypted even if automatic encryption is enabled. Added a new --maxMegabytesPerSecond argument that can be used to impose a limit on the rate at which the LDIF file may be written to disk.

* Updated the import-ldif tool to add new --promptForEncryptionPasshprase and --encryptionPassphraseFile arguments that can be used to provide a user-supplied passphrase for use in accessing the contents of an encrypted LDIF export. The --isEncrypted and --isCompressed arguments are no longer necessary, as the tool can automatically detect encryption and compression (although those arguments are still available to preserve backward] compatibility), and it can automatically identify the correct key for exports encrypted with a key obtained from an encryption settings definition or an internal topology key.

DS-14242,DS-16519,DS-35778

Made several improvements to the verify-index tool:

* Fixed an issue that could cause the tool to report an inaccurate estimate for the number of records to process.

* Fixed an issue that could cause the tool to perform more processing than necessary when verifying JSON or composite indexes without the --clean argument.

* Improved performance when verifying composite indexes.

* Improved validation for equality indexes configured with an additional filter.

* If any validation errors are encountered, it is now much easier to access the details of those validation errors, and the error messages are much clearer.

DS-14248

Changed the default settings for the Directory Changelog to include all entry attributes during a delete so that matching can be more reliable.

DS-15146

Updated the SMTP account status notification handler to add support for extracting email addresses from JSON attributes, including an optional filter that could be used for things like only sending messages to verified and/or preferred addresses.

DS-15223,DS-35895

Added the ability to configure data encryption during setup using a randomly generated key, a key generated from a user-supplied passphrase, or a key obtained from an export of another server's encryption settings database. When setting up multiple instances, providing the same encryption passphrase to each instance will ensure that all instances have the same encryption key.

The encryption-settings tool has also been updated to allow creating encryption settings definitions from a passphrase, to allow providing a description when creating a new encryption settings definition, and to record a create timestamp for new definitions. It is now possible to create ciphers that use the Galois Counter Mode (GCM) cipher mode (for example, using a cipher transformation of "AES/GCM/PKCS5Padding") for authenticated encryption. Definitions created with with just a cipher algorithm but no transformation will now use stronger settings.

The default encryption settings export format now provides stronger encryption. Newer server instances should be able to import encryption settings exported from other servers without issue. When exporting encryption settings for import into older servers, use the new --use-legacy-export-format argument.

DS-16508

Updated the dsconfig list subcommands to list objects of all complexity levels rather than requiring the --advanced flag to list advanced and expert objects.

DS-17158

Updated the server to add support for new "reject unindexed search" and "permit unindexed search" request controls, which make it possible to have more fine-grained control over when the server should process an unindexed search.

DS-17197

The dsreplication command's remove-defunct-server and cleanup-local-server sub-commands have been replaced by a new command called remove-defunct-server. The new command supports all of the functionality that was provided by the sub-commands.

DS-17883

Added an ldap-debugger tool that acts as a simple LDAP proxy between a client and a directory server and decodes all requests and responses that pass through it.

DS-17891

Added a new manage-certificates tool that can be used to perform a number of functions related to TLS certificate management.

DS-18203

Addressed an error that occurs if a password policy DN is modified during an operation that is dependent on that password policy.

DS-18204

Fixed a defect where a web application extension's base context path could be set to "/" with no name.

DS-18222

Made the sending of heartbeats between replication servers more efficient.

DS-35494

Fixed an intermittent deadlock in assured replication.

DS-35523

The update tool now enforces specification of a new product license when updating to a new major version. The license can be specified using the --licenseKeyFile command-line options, or by copying the license file to the top-level directory of the server package used to perform the update. Request a license key through the Ping Identity licensing website https://www.pingidentity.com/en/account/request-license-key.html, or contact sales@pingidentity.com.

DS-35528

Addressed an issue in "dsreplication enable/initialize" that prevented servers from some previous versions (5.2.0.5 and earlier and 6.0.0.*) from initializing newer servers. Servers from these prior versions can now be used to enable replication with current versions of the server.

DS-35534

Added two database checkpoints to be performed in the backend prior to each online backup. If the backup is restored, this limits the database recovery time when the database environment is opened for the first time, which significantly improves the server startup time. Also, incremental backups will not contain redundant files that are unchanged from the base backup.

DS-35536

Support for the IBM JDK has been retired.

DS-35537

Updated how indexes respond to corrupt records. An index with a corrupt record no longer goes offline and degrades its server. The index instead raises an alarm and continues to process requests. Such an index should be rebuilt, but it will continue to be used and will return the correct results for unaffected keys.

DS-35538

Updated JZlib from version 1.0.7 to 1.1.3 to address an ArrayIndexOutOfBoundsException that could occur when replication servers disconnected during a scheduled garbage collection.

DS-35549

Eliminated a misleading error message triggered by sorting entries during a search.

DS-35551

Updated the Directory Server to require a minimum Berkeley DB Java Edition version of 7.5.11.

Builds of the server that ship with the Berkeley DB JE jar file actually include a pre-release jar file for version 7.5.12 that fixes an issue that could result in lock conflicts internally within JE under heavy concurrent load in an I/O-bound database. Organizations that receive a Directory Server build that does not include the Berkeley DB Java Edition jar file are recommended to contact Oracle support to obtain this fix under their own JE support contract.

DS-35558

Fixed an issue where a JSON object or attribute could not be deleted if it contained a field with two indexed values differing only in case.

DS-35571

In replicated environments, updating attributes with a very large number of attribute options is now much faster. There is a potential for a 2X increase in storage for these entries, but updating an entry with 1000 attribute options is now 100 times faster.

DS-35576

Updated the JMX connection handler's monitor provider so that when a JMX connection is closed, it is removed from the list of established connections. After a JMX client disconnects, it may take the server a few minutes to detect the closure and update the monitor.

DS-35580

Updated the Frequently Accessed Entries Cache to eliminate cache lock contention when the server is starting, and increased parallelism in general. Also, updated the cache to penalize frequently modified entries to improve garbage collection and reduce heap fragmentation.

DS-35581

Updated the server to include an instance of the Periodic Stats Logger Plugin that is enabled out-of-the-box to aid in diagnosing support issues. The "Historical Stats Logger" plugin will log performance statistics to logs/monitor-history/historical-dsstats.csv every five minutes. This works in concert with the "Monitor History" plugin, which logs the full contents of cn=monitor to logs/monitor-history every five minutes. The tail of this csv file is automatically included in the output generated by collect-support-data.

DS-35582

Added an "Expensive Write Operations Access Logger" that adds detailed timing information to logs for add, delete, modify, or modify-dn operations that take longer than the configured threshold (one second by default). Log messages include information about the most expensive phases of operation processing, which can help diagnose the cause of performance outliers.

DS-35583

Fixed a defect where configuring a Directory server on a Windows machine with a space in the home directory pathname would cause server setup to fail.

DS-35593

Removed the max-passes option from the Periodic GC Plugin. This setting was sometimes necessary in Java 1.6, but setting it higher than 1 now just increases the amount of time it takes the plugin to complete a full garbage collection.

DS-35601

Added a new Monitor Entry for SSL Cipher Suite and Protocol information. It is available under cn=SSL Context,cn=monitor.

DS-35616

Fixed a very rare race condition with the Frequently Accessed Entry Cache which could lead to an index being marked as degraded and requiring a rebuild.

The problem is unlikely to happen outside of testing environments since it requires modifying a single entry over 1000 times per second across multiple servers concurrently.

DS-35648

Added a missing double-quote to bat/transform-ldif.bat, which prevented the command from being invoked successfully on Windows systems.

DS-35659

Enhanced the ACI handling code to reduce contention when updating the set of active ACIs. In systems that have a very high number of entries that include ACIs, this enhancement significantly improves the throughput of operations that add and delete ACIs.

DS-35694,DS-36189

Added a new Cleanup Expired PingFederate Persistent Access Grants Plugin. An instance of this plugin can be created to automatically delete expired persistent access grant entries that were created by PingFederate. This is preferable to PingFederate's cleanup process since it can be run on multiple servers and throttles the background delete operations to prevent impacting client traffic.

DS-35709

Changed enable-sub-operation-timer on the Global Configuration to be true by default. This exposes operation timing information in the Sub-Operation Timing Monitor and any Operation Timing Access Log Publishers that have been configured. Enabling this tracking has about a 3% impact on operation throughput and latency, which will not be noticeable in most deployments and is an acceptable tradeoff for understanding where operation processing time is spent. However, it can be explicitly set to false to turn this tracking off.

DS-35727,DS-35728

Updated setup to include key usage, extended key usage, and subject alternative name extensions in the self-signed certificates that it generates.

DS-35750

Added support for multiple client connection policies for sensitive attributes. Support for different sensitive attributes per client requires the use of multiple client connection policies with the same names on the Directory Server and the Proxy Server. When a client request is processed by a Proxy Server, the Directory Server looks for a policy in its own configuration with the same name as the one in the Proxy Server. The Directory Server then uses this policy rather than the one associated with the Proxy Server's connection.

DS-35790

Addressed an issue where an index key could incorrectly be reported as exceeding the index-entry-limit after one billion entries had been imported or added to the directory server. The directory server does not need to contain one billion entries at the same time to be affected by this issue since the entry ID will always increase for each added entry even if entries are deleted. Environments that have experienced this issue should export and reimport their data after applying this patch.

DS-35797

The dsreplication "enable" and "initialize" commands now support adding or initializing a server using a topology file in non-interactive mode. The topology file must contain the list of servers that are already in the topology and may be obtained by running the "manage-topology export" command. An LDAP connection to the first available server in the list is used to add or initialize the new server into the topology.

DS-35797

The dsreplication "enable" and "initialize" commands now support adding or initializing a server using a topology file in non-interactive mode. The topology file must contain the list of servers that are already in the topology, and may be obtained by running the "manage-topology export" command. An LDAP connection to the first available server in the list (preferably in the same location as the new server) is used to add or initialize the new server into the topology.

DS-35868

The create-systemd-script command now suggests placing the script created in "/etc/systemd/system."

DS-35869

Updated the GSSAPI SASL mechanism handler to support alternate authorization identities, to support the "dn:" and "u:" formats for those authorization identities, and to allow the use of different identity mappers for authentication and authorization identities.

DS-35898

Updated the LDIF export task so that the server will now execute the export-ldif command in a separate process. This is safer than the former export task mechanism because it ensures that the exported LDIF file will reflect the contents of the backend at a specific point in time (the time the export-ldif process opened the database).

DS-35933

Fixed a password policy issue that could arise if a user authenticated with a password that was stored with a deprecated password storage scheme. If the password policy for that user is configured with a force-change-on-reset value of true, then the internal update used to re-encode that password would have incorrectly been classified as an administrative reset and the user would have been required to choose a new password after the next successful bind.

DS-35990

Provided the means to request that the server dynamically reload the certificate key and trust stores used by all HTTP connection handler instances that provide support for HTTPS. The request can be made using a new reload HTTP connection handler certificates task, the reload-http-connection-handler-certificates tool, or programmatically from a Server SDK extension using the ServerContext#reloadHTTPConnectionHandlerCertificates method.

DS-35995

The task that sets the generation ID now waits for the replica to be connected before preceding. This should prevent errors such as "The generation ID could not be reset for domain dc=example,dc=com because it is NOT connected to the replication."

DS-35997

Enhanced schema validation. Backends with compacted attribute types will not open if any of the attribute types are missing from the schema.

DS-36025

Added a close-connections-when-unavailable property to the LDAP Connection Handler configuration. This allows a connection handler to be closed whenever the server sets an unavailable alert type, such as when backend data is unavailable. This should trigger clients to failover to another server. When the unavailable alert type is cleared, the connection handler is started again. When using this configuration setting, we recommend using two connection handlers: one for client traffic, with this option set to true, and one for administration and monitoring, with this option set to false. This allows the server to be visible to administrators but not to clients.

DS-36054

Added an encrypt-file tool that can encrypt and decrypt data with a user-supplied passphrase, an encryption settings definition, or a topology key shared among server instances. It includes support for decrypting the content in encrypted backups, LDIF exports, and log files.

DS-36070

Fixed an issue with compressed logging that could leave some data buffered in memory and not actually written out to disk until the logger is closed.

DS-36075

Updated tools that interact with log or LDIF files to support reading from input files that are compressed and encrypted and writing to compressed and encrypted output files.

DS-36088

In addition to specifying an exact set of desired cipher suites for the LDAP and HTTP Connection Handlers, administrators can now specify inclusions to, or exclusions from, the set of cipher suites selected by the server.

DS-36092

Updated the rebuild-index tool so that the bulkRebuild argument's usage description and error message make it clear that it cannot be used when running the tool as a task or while server is online.

DS-36093

Added support for TLS1.2 with STARTLS to connect to an SMTP server.

DS-36093

Added support for TLS1.2 with STARTLS to connect to SMTP server

DS-36111

Added support for backup retention. When performing a backup, it is now possible to specify a minimum number of backups to retain or a minimum age of backups to retain. If either or both are specified, and if the new backup completes successfully, then any previous backups in the same directory that do not satisfy that criteria will be removed.

The remove-backup tool has also been updated to allow specifying the number of backups to retain or the minimum age for backups to retain as an alternative to providing the specific backup ID for the backup to remove. If retention criteria is specified, then any backups in the target directory that do not satisfy that criteria will be removed.

DS-36230

Improved the server's handling of DNs and RDNs that contain characters whose UTF-8 encodings require more than two bytes.

DS-36239

Fixed an issue to address null addresses associated with internal connections when used in conjunction with address-based ACI evaluations, such as IP and DNS bind rules.

DS-36272

Fixed an issue that could cause certain special characters in LDAP URLs to be doubly encoded.

DS-36312

Fixed an issue that could cause the server to incorrectly classify some search requests as unindexed.

DS-36328

Updated the server to reduce contention when converting between strings and the bytes that comprise those strings.

DS-36339

The error message for a null changelog token when the current server is the Master has been removed

DS-36360

Increased the default size of the queue used to hold alert notifications so they can be asynchronously processed by a background thread. This makes it less likely that the queue will become full if many alerts are generated in a short period of time, which would cause subsequent attempts to generate alerts to block while the server catches up. Also updated the server to log a message when the queue becomes full so that administrators will be aware of the problem and will have suggestions for addressing it.

DS-36377

Fixed an issue where a configuration change to enable a Delegated Administrator could be incorrectly rejected after a configuration change to the parent Delegated Admin Resource Type.

DS-36403

Changed server behavior so it will now mark a JSON index as corrupt if it can't find a key, instead of having the operation fail.

DS-36439

Fixed a rare race condition that would cause "dsreplication enable" or "dsreplication initialize" to fail. This issue has only been seen in automated testing environments with resource-constrained virtual machines.

DS-36545

Added a sanitize option to the Monitor History Plugin that, if enabled, will redact the small amount of potentially personally identifiable information that could appear in search filters and LDAP DNs within the monitor. This makes it easier to share the monitor history files with the support team in secure environments.