Upgrade Considerations

Important considerations for upgrading to this version of the Directory Server:

Important:

If you plan to upgrade servers using a mixed-version environment where one version is earlier than 7.0 and some of the servers are still using the admin backend while others have been updated to the topology registry, do not attempt to make size changes to the topology. You cannot remove any existing servers (using dsreplication disable) or add new servers (using dsreplication enable) when in this transitional state of partially-updated servers. When a topology has been completely migrated to a 7.0 or later version with the topology registry, changes to the topology size are allowed, even in mixed-version environments (for example, mixed 7.3 and 8.3).

  • To ensure correct search results with Delegated Admin, disable client caching by updating the Delegated Admin HTTP Servlet Extension to return response headers, and then stop and restart the server, as follows:

    dsconfig set-http-servlet-extension-prop --extension-name "Delegated Admin" --set "response-header:Cache-Control: no-cache, no-store, must-revalidate" --set "response-header:Expires: 0" --set "response-header:Pragma: no-cache"

What's New

These are new features for this release of the Directory Server:

  • Introduced a new capability called Server Profiles to manage PingDirectory instances following the DevOps principle of infrastructure-as-code. Administrators can export the configuration of a PingDirectory instance to a directory of text files called a Server Profile, track changes to these files in version control like Git, and install new instances of PingDirectory or update existing instances of PingDirectory from a Server Profile. Server Profiles support variable substitution in order to remove the settings unique to each pre-production or production environment from the Server Profile that is stored in version control.

  • Several improvements to support highly automated or orchestrated environments. Replication management tools have been improved to support typical workflows such as: simultaneous automated initialization of multiple instances such as during a scale out operation; automated initialization of instances despite other instances being unavailable such as during a replacement of a failed instance; automated planned removal of an instance such as during a scale down. Additionally, an HTTP status endpoint has been added to report overall instance health and availability to a cluster orchestrator like Kubernetes or to a network load balancer like AWS Network Load Balancer.

  • New features for delegated administration: using REST Resource Types, administrators can delegate management of new resource types, like groups and organizations, in addition to the user management features available previously. Using Delegated Access Rights, administrators now have independent controls over delegation of search, view, update, and create per resource type.

  • New plugin for pass-through authentication to PingOne for Customers. Pass-through authentication allows passwords to be managed by users and administrators in PingOne for Customers while still supporting legacy LDAP application connections on-prem.

  • New features for data encryption in transit and at rest: added support for TLS 1.3, ability to encrypt and automatically decrypt sensitive files such as tools.properties and keystore pin files using the server data encryption keys, and the ability to more easily and securely separate master keys from data encryption keys by protecting the server encryption settings database using either Amazon Key Management Service (AWS KMS) or HashiCorp Vault.

  • Improved replication resiliency to extended network partitions or downtime. The retention period for the replication changelog now supports a disk usage threshold in addition to an age limit for change retention. In combination, this means changes under typical load can be retained for longer than the age limit as long as the disk usage threshold is not exceeded.

  • Added support for Amazon Corretto JDK 8, Windows Server 2019, Red Hat Enterprise Linux 7.6, CentOS 7.6, Amazon Linux 2, and Docker 18.09.0 on Ubuntu 18.04 LTS.

Known Issues/Workarounds

The following are known issues in the current version of the Directory Server:

  • When dsreplication is run to add a server to the topology using another node that is not the topology master, it may fail with the following error:

    "Error updating replication configuration on base DN dc=example,dc=com of server 'ds3' (ldaps://localhost:3636). See /Users/<name>/installs/7.2/s3/logs/tools/dsreplication.log for a detailed log of this operation. Details: A communication problem occurred while contacting the server: The connection to server localhost:3389 was closed while waiting for a response to an add request AddRequest(dn='cn=dc_example_dc_com,cn=domains,cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config', attrs={Attribute(name=objectclass, values={'top', 'ds-cfg-replication-domain'}), Attribute(name=cn, values={'dc_example_dc_com'}), Attribute(name=ds-cfg-server-id, values={'11443'}), Attribute(name=ds-cfg-base-dn, values={'dc=example,dc=com'})}): A request sent on this client connection caused an internal error in the server. This connection will be terminated."

    The workaround for this issue is to use the topology master for the --host1 parameter of dsreplication to add the new server into the topology.

Resolved Issues

The following issues have been resolved with this release of the Directory Server:

Ticket ID Description
PDSTAGING-570,DS-38334

The following enhancements were made to the topology manager to make it easier to diagnose the connection errors described in PDSTAGING-570:

- Added monitoring information for all the failed outbound connections (including the time since it's been failing and the last error message seen when the failure occurred) from a server to one of its configured peers and the number of failed outbound connections.

- Added alarms/alerts for when a server fails to connect to a peer server within a configured grace period.

PDSTAGING-570,DS-38344

The topology manager will now raise a mirrored-subtree-manager-connection-asymmetry alarm when a server is able to establish outbound connections to its peer servers, but those peer servers are unable to establish connections back to the server within the configured grace period. The alarm is cleared as soon as there is connection symmetry.

PDSTAGING-570,DS-38335

The dsreplication tool has been fixed to work when the node being used to enable replication is currently out-of-sync with the topology master.

DS-6401

Added a target-database-size parameter to the Changelog Backend and the Replication Server configuration objects to allow the corresponding changelogs to grow on disk beyond the purge delay up to the specified total disk size.

It is possible that the changelog database size on disk exceeds this configured value since changes are never purged before the configured purge delay

A new "Changelog Database Target Size (Percent)" gauge is also included, which will raise an alarm if a changelog grows on disk above the specified limit.

So that the server can more easily achieve the target database disk size, the default log file size for both database environments has been reduced from 100MB to 10MB.

DS-15734

Added a cipher stream provider that can be used to protect the contents of the encryption settings database with a key from the Amazon Key Management Service.

DS-17630

Enabled secure JMX connections. Clients must specify the truststore when running applications, and possibly the type and password for the truststore.

DS-18060

Added an HTTP servlet extension that can be used to retrieve the server's current availability state. It accepts any GET, POST, or HEAD request sent to a specified endpoint and returns a minimal response whose HTTP status code may be used to determine whether the server considers itself to be AVAILABLE, DEGRADED, or UNAVAILABLE. The status code for each of these states is configurable, and the response may optionally include a JSON object with an "availability-state" field with the name of the current state.

Two instances of this servlet extension are now available in the default configuration. A request sent to /available-state will return an HTTP status code of 200 (OK) if the server has a state of AVAILABLE, and 503 (Service Unavailable) if the server has a state of DEGRADED or UNAVAILABLE. A request sent to the /available-or-degraded-state will return an HTTP status code of 200 for a state of AVAILABLE or DEGRADED, and 503 for a state of UNAVAILABLE. The former may be useful for load balancers that you only want to have route requests to servers that are fully available. The latter may be useful for orchestration frameworks if you wish to destroy and replace any instance that is completely unavailable.

DS-36474

Updated the ldapdelete command-line tool to improve robustness and add features. Some of the new features include support for client-side subtree delete, deleting entries that match search filters, following referrals, writing failures to a rejects file, rate limiting, and support for a variety of additional controls.

DS-36685

Improved the diagnostic message the server returns when rejecting a proxied authorization attempt because the target account's password policy state does not permit that user to authenticate.

DS-37063

Added a plugin that can allow users to authenticate to the Directory Server with credentials from corresponding accounts in the PingOne for Customers service.

DS-37063,DS-38012,DS-38497

Updated the pass-through authentication plugin to add a bind-dn-pattern configuration property that allows constructing the DN of the remote user from information in the local user entry. Alternately, if the remote server supports simple authentication with a bind DN value that is not actually a valid LDAP distinguished name, this property can be used to supply that identifier. For example, when passing through authentication to Microsoft Active Directory, this property can be used to construct a bind DN that is actually the user principal name (UPN) for the remote account.

Also, updated the pass-through authentication plugin to add an included-local-entry-base-dn configuration property that can be used to indicate which local entries are eligible for pass-through authentication. By default, pass-through authentication is automatically enabled for all users contained in any public backend, but this property can be used to restrict that set of users without the need to define a request criteria. This change also ensures that the server no longer attempts to pass through authentication attempts for root users or topology administrators by default (although if that ability is desired, it can be re-enabled by adding "cn=config" as an included-local-entry-base-dn value).

DS-37164

Replication now removes references to obsolete replicas, in both LDAP and in the BDB database. A replica is obsolete when it has been disabled and all changes are older than the replication purge delay.

DS-37363

Updated the server to make the replication missing changes state persist across restarts. If a server is offline for longer than the configured purge delay, then replication cannot automatically bring the server back in-sync with the rest of the topology. To avoid serving stale data, the server enters lockdown mode when it has missed changes. Prior to this change, restarting the server would incorrectly clear this missing changes state, and it would not enter lockdown mode, which could lead to it serving stale data. Now the server must be reinitialized either from a recent backup or by using "dsreplication initialize" to clear the missing changes state.

DS-37430

Added logging for DNS lookups that take longer than a warning threshold. The default warning threshold is 10 seconds. Added the DNS Resolution monitor to track DNS lookup speed.

DS-37617

HTTP Connection Handlers now accept client-provided correlation IDs by default. To adjust the set of HTTP request headers that may include a correlation ID value, change the HTTP Connection Handler's correlation-id-request-header property.

DS-37757

Fixed an issue that could cause entryUUID mismatches on replicas configured to automatically use entryUUID as the naming attribute for add requests matching a given set of connection or request criteria.

DS-37805

Updated the server to better sanitize information included in diagnostic messages included in responses to clients. In some cases (for example, in the event of a unique attribute conflict), a response diagnostic message could disclose the existence of another entry in the server.

DS-37839

Make Fingerprint Certificate Mapper and Subject DN to User Attribute Certificate Mapper disabled by default on fresh installations. This will not affect upgrades from installations where these mappers are enabled.

DS-37842

Fixed an issue that allowed a modify operation to alter an entry in a way that left it without one or more superior object classes.

DS-37959

Added support for insignificant configuration archive attributes.

The configuration archive is a collection of the configurations that have been used by the server at some time. It is updated whenever a change is made to data in the server configuration, and it is very useful for auditing and troubleshooting. However, because the entries that define root users and topology administrators reside in the configuration, changes to those entries will also cause a new addition to the configuration archive. This is true even for changes that affect metadata for those entries, like updates to the password policy state information for one of those users. For example, if last login time tracking is enabled for one of those users, especially with high-precision timestamps, a new configuration may be generated and added to the configuration archive every time that user authenticates to the server. While it is important for this information to be persisted, it is not as important for it to be part of the server's configuration history.

This update can help avoid the configuration archive from storing information about updates that only affect this kind of account metadata. If a configuration change only modifies an existing entry, and if the only changes to that entry affect insignificant configuration archive attributes, then that change may not be persisted in the server's configuration archive.

By default, the following attributes are now considered insignificant for the purpose of the configuration archive:

* ds-auth-delivered-otp * ds-auth-password-reset-token * ds-auth-single-use-token * ds-auth-totp-last-password-used * ds-last-access-time * ds-pwp-auth-failure * ds-pwp-last-login-ip-address * ds-pwp-last-login-time * ds-pwp-password-changed-by-required-time * ds-pwp-reset-time * ds-pwp-retired-password * ds-pwp-warned-time * modifiersName * modifyTimestamp * pwdAccountLockedTime * pwdChangedTime * pwdFailureTime * pwdGraceUseTime * pwdHistory * pwdReset

DS-37960

The Delegated Admin configuration has changed significantly. Delegated Admin Resource Types were removed and replaced by REST Resource Types. Delegated Administrators and Delegated Group Administrators were removed and replaced by Delegated Admin Rights and Delegated Admin Resource Rights. Previous configurations are converted to the new configuration definitions by the update tool when the server is updated.

DS-38021

Enabled assured replication by default for all add, delete, and modify DN operations. Enabled assured replication by default for all modify operations that alter passwords or key password policy state attributes.

The server will now wait (up to a maximum of one second) for these types of changes to be replicated to all available local servers before returning the response to the client. This can help avoid issues that may arise if a client sends a write request, and then immediately sends another request that depends on that previous request. If the two requests are routed to different servers, then the second operation may fail or yield an unexpected result if the change from the first request has not yet been replicated.

This change will only take effect for new installations. It will not apply to existing installations that are updated to the new release.

DS-38050

Updated the server to support encrypting the contents of the PIN files needed to unlock certificate key and trust stores. If data encryption is enabled during setup, then the default PIN files will automatically be encrypted.

Also, updated the command-line tool framework so that the tools.properties file (which can provide default values for arguments not provided on the command line), and passphrase files (for example, used to hold the bind password) can be encrypted.

DS-38072

Updated the server to enable TLSv1.3 by default on JVMs that support it (Java 11 and higher).

DS-38077

Servers that are removed from replication with the "dsreplication disable" command are now also removed from the topology when the last non-schema domain is disabled. This allows the state of the servers after the disable to be closer to the pre-enabled state.

DS-38083

Fixed ordering of consent-service-cfg.dsconfig commands so that bearer token authentication is enabled after its dependency, unprivileged consent.

DS-38085

Fixed an issue in the installer where the Administrative Console's trust store type would be incorrectly set if it differed from the key store type.

DS-38090,DS-38564,DS-38567

The response header used for correlation IDs may now be set at the HTTP Servlet Extension level using the correlation-id-response-header configuration property. If set, this property overrides the HTTP Connection Handler's correlation-id-response-header property.

DS-38109

Added the --skipHostnameCheck command line option to the setup script, which bypasses validation of the provided host name for the server.

DS-38190

Fixed a problem that could cause a negative etime to appear in the access log when using assured replication.

DS-38193

Updated the behavior the server exhibits if an attribute type definition is removed from the schema while it is still referenced by a local DB backend's compaction dictionary.

In an attempt to minimize the amount of disk space and memory needed to store information in the database, the server compacts the data in several ways. One of its compaction techniques is to reference attribute types by tokens rather than their full name. It maintains a dictionary of these tokens so that it can quickly translate between an attribute type and its corresponding token. If an attribute has been used in at least one entry in the backend since the last LDIF import, then this dictionary should include a token for that attribute type.

On startup, the server will read this compaction dictionary into memory. Previously, if it encountered a reference to an attribute type that had been used in the backend but is no longer defined in the schema, it would abort the startup process. This behavior has been changed so that it will instead generate an administrative alert to warn administrators of the problem and provide information about how to address the issue, but it will no longer abort the startup process. The server will also generate an administrative alert if it encounters an entry whose encoded representation includes a token that is associated with an undefined attribute type.

In addition, the server has also been updated so that it will no longer permit attribute types or object classes to be removed from the schema if they are referenced in a compaction dictionary.

DS-38202

Fixed an issue that could cause an error during an LDIF export of a data set with a large number of non-leaf entries. In such cases, the data is written to multiple files that are merged at the end of the export process. If the LDIF export was encrypted with a passphrase or an encryption settings definition, the merge process could fail, leaving the export spread across multiple files instead of aggregated into a single file.

This issue did not affect the usability or integrity of the export data. It could still be imported, although the administrator would need to list each of the export files in the correct order when performing the import.

DS-38205

Addressed potential parsing errors in the periodic stats logger when the server is deployed in a non-English locale.

DS-38272

Fixed a Windows stop-server.bat issue where locales using commas for decimal separators could not shut down.

DS-38273

Fixed an issue where changes to a dynamic group's member URL sometimes did not take effect until the next server restart.

DS-38283

Updated ACI processing for modify DN requests. The "export" and "import" rights are no longer required if the superior DN is provided, but has not changed.

DS-38291

Fixed an issue with the client-side validation properties that the haystack password validator would return in a get password quality requirements extended response.

DS-38336

Updated the result code map to allow overriding the default result code that the server returns when a client tries to perform a password-based bind as a user who doesn't have a password.

DS-38403

Fixed an issue that could prevent certain types of initialization failures from appearing in the server error log by default.

DS-38415,DS-38418,DS-38419,DS-38420

Addressed several issues with the pass-through authentication plugin.

USE SEPARATE CONNECTIONS FOR SEARCH AND BIND REQUESTS

If the plugin is configured with a search filter pattern, then it may perform a search to find the entry in the external server that corresponds to the entry for the local user that is trying to bind. In such cases, search requests may have been issued over the same connections that were also been used to process bind operations. The change in authorization identity resulting from those bind attempts may interfere with the ability to perform the searches. The plugin has been updated to ensure that search and bind requests are now issued over separate connections.

ALLOW RETRY ATTEMPTS WITH A SINGLE EXTERNAL SERVER

If the plugin is configured with multiple external servers, then it can use some or all of those servers in a pass-through authentication attempt. If a search or bind attempt fails against the first server, and if that failure indicates that there may have been a problem with the server or the connection to it, then the plugin would have re-tried the operation in other servers until the attempt succeeds, the attempt fails in a way that does not indicate a problem with the server or the connection, or all servers have been tried. However, if only a single external server had been configured, then no retry attempt would have been made. The plugin has been updated so that if it is only configured with a single external server, and if a failure is encountered while communicating with that server that may benefit from retrying that operation, then the plugin will attempt to establish a new connection to that server and retry the operation.

UTILIZE ALL CONFIGURED EXTERNAL SERVERS

If PingDirectory Server is configured with a location, then the pass-through authentication plugin will use that information to determine the order in which the external servers should be accessed. It will first attempt external servers in the same location as PingDirectory Server, followed by servers in the most preferred failover location, the second-most preferred failover location, and so on. However, the plugin might have used external servers that did not have a location assigned, or that were assigned to a location that is not one of PingDirectory Server's preferred failover locations. The plugin has been updated to ensure that these servers may be used, albeit with a lower priority than the other servers.

DS-38415,DS-38418,DS-38419,DS-38420 (cont.)

IMPROVE VISIBILITY OF PLUGIN PROCESSING RESULTS

The plugin offered very little information that could help an administrator troubleshoot problems with pass-through authentication processing. Some types of operations could be investigated by enabling debug logging with an appropriate scope, but no information about the pass-through authentication processing would appear in the PingDirectory Server access log. The plugin has been updated to add information about its processing to the bind operation's access log message, including the ultimate success or failure of the pass-through authentication attempt, the result of user mapping, and whether the local user's password was updated. Further, the plugin now makes more information about its internal processing available through the server's debug logging facility.

DS-38421

Updated the file retention recurring task to no longer log an informational message if there are no log matching files to delete.

DS-38430

To efficiently export entries from a backend with a large number of non-leaf entries, the export-ldif command produces multiple intermediate LDIF files that are merged after all entries have been processed. The tool now skips merging these files if sufficient disk space is unavailable to accommodate the intermediate files and the final merged file. The import-ldif command can accept multiple files as input, so merging the files is not essential.

DS-38512

Added a cipher stream provider that can be used to protect the contents of the encryption settings database with a secret passphrase obtained from a HashiCorp Vault instance.

DS-38524

Fixed a problem in which a member could not be added to a group via SCIM unless the group's object class was groupOfUniqueNames.

DS-38525

Made changes to reduce potential lock conditions in proxy replication LDAP health checks.

DS-38550

Fixed an issue in which backups of the encryption settings database could be encrypted with a key from the encryption settings database.

DS-38643

The "dsreplication disable" command now correctly removes replica IDs (ds-cfg-replication-domain-server-id values) from the topology data when a subset of the replication domains is disabled.

DS-38663

Updated performBackendDeregistrationProcessing and performBackendRegistrationProcessing to ignore disabled notification managers instead of throwing a null error.

DS-38670

Fixed a bug where the startIndex value for SCIM requests would be incorrect if the used LDAPSearch element had more than one baseDN defined in the scim-resources XML file.

DS-38722

Fixed an issue where a constructed virtual attribute could not be configured for an attribute that was marked as SINGLE-VALUE in the LDAP schema.

DS-38737

Fixed an issue where inter-server bind requests would fail if the cipher used reported a maximum unencrypted block size of 0.

DS-38739

Updated the encrypt-file tool to display a notice recommending the use of the --decompress-input argument when decrypting a file that also appears to be GZIP-compressed.

DS-38839

The Amazon Corretto JRE is now supported. Prior to being supported, the Amazon Corretto JRE resulted in a warning that the JRE was "unrecognized and is likely untested and unsupported."

DS-38849

Added an indent-ldap-filter tool that can make it easier to visualize the structure and components of a complex search filter.

DS-38864

Changed the default value of the HTTP Configuration property include-stack-traces-in-error-pages from 'true' to 'false'. Disabling this property prevents information about exceptions thrown by servlet or web application extensions from being revealed in HTTP error responses.

DS-38873

Internal connections created by HTTP requests are now associated with one of the configured client connection policies. A client connection policy may be selected using simple client connection criteria matching the client address, the user performing the request, and the protocol "HTTP/1.1". This change affects the following HTTP interfaces: SCIM, Directory REST API, Consent API and Delegated Admin API.

DS-38874

Updated the server to prevent creating virtual attributes that use the "aci" or "ds-cfg-global-aci" attribute types. Also, updated the server to prevent creating virtual attributes that use the "member" or "uniqueMember" attribute types unless the virtual attribute is one that will provide the membership list for a virtual static group.

Virtual attributes cannot be used to define access control rules or assign static group membership. Previously, the server silently ignored any access control rules or static group members defined through virtual attributes, which may have caused an administrator to mistakenly believe that they were in effect.

DS-38892

Fixed an issue that could cause the server to encounter an internal error when processing a set subtree accessibility extended operation against an empty backend.

DS-38893

Fixed an issue that interfered with assigning privileges using a mirror virtual attribute. If the values to mirror in the ds-privilege-name attribute were contained in another entry, then the privileges would have only been granted if the source attribute could be retrieved by unauthenticated clients.

DS-38897,DS-38908

Fixed two issues in which the server could have exposed some clear-text passwords in files on the server file system.

* When creating an encrypted backup of the alarms, alerts, configuration, encryption settings, schema, tasks, or trust store backends, the password used to generate the encryption key (which may have been obtained from an encryption settings definition) could have been inadvertently written into the backup descriptor. This problem does not affect local DB backends (like userRoot), the LDAP changelog backend, or the replication database.

* When running certain command-line tools with an argument instructing the tool to read a password from a file, the password contained in that file could have been written into the server's tool invocation log instead of the path to that file. Affected tools include backup, create-initial-config, create-initial-proxy-config, dsreplication, enter-lockdown-mode, export-ldif, import-ldif, ldappasswordmodify, leave-lockdown-mode, manage-tasks, manage-topology, migrate-ldap-schema, parallel-update, prepare-endpoint-server, prepare-external-server, realtime-sync, rebuild-index, re-encode-entries, reload-http-connection-handler-certificates, reload-index, remove-defunct-server, restore, rotate-log, and stop-server. Other tools are not affected. Also note that this only includes passwords contained in files that were provided as command-line arguments; passwords included in the tools.properties file, or in a file referenced from tools.properties, would not have been exposed.

In each of these cases, the files would have been written with permissions that make their contents only accessible to the system account used to run the server. Further, while administrative passwords may have been exposed in the tool invocation log, neither the passwords for regular users, nor any other data from their entries, should have been affected. We have introduced new automated tests to help ensure that such incidents do not occur in the future.

We recommend changing any administrative passwords you fear may have been compromised as a result of this issue. If you are concerned that the passphrase for an encryption settings definition may have been exposed, then we recommend creating a new encryption settings definition that is preferred for all subsequent encryption operations, exporting your data to LDIF, and re-importing so that it will be encrypted with the new key. You also may wish to re-encrypt or destroy any existing backups, LDIF exports, or other data encrypted with a compromised key, and you may wish to sanitize or destroy any existing tool invocation log files that may contain clear-text passwords.

DS-38913

Added a set of message types to Trace Log Publishers that records events related to access token validation.

DS-38957

Fixed an issue that would throw an exception when trying to delete an entry containing uncached attributes if the LDAP changelog was enabled and using reversible form.

DS-39086

Removed the version information page from the docs/build-info.txt endpoint. This information is now available in build-info.txt, which is located in the root directory.