What's New

These are new features for this release of the PingDirectory Server:

  • Improvements to Datastore’s Native JSON Attribute Support. The previous release of the Datastore included native support for compactly storing JSON objects in LDAP attributes, and for evaluating filters to match on fields inside those JSON objects. This release adds support for enforcing constraints on the fields that may be included in JSON objects, for indexing field values for improved search performance and flexibility, and for tokenizing commonly-used string values to further reduce the data footprint.

  • New Pasword Validators. The Datastore now includes the Haystacks Password Validator, based on the Gibson Research Corporation Password Haystacks concept, and the Commonly-Used Passwords Dictionary Validator, which ensures that a proposed password is not one of 10,000 commonly used passwords.

  • Can Request an entryUUID rdn Value when Adding an Entry. When adding a new entry to the server, the client can now request that the server-generated entryUUID be used as the RDN attribute for the entry. This improves privacy by ensuring that the entry DN will not include any personally-identifying information, and it is a convenience for application developers by eliminating the potential need for modify DN operations.

  • Implemented a Virtual Attribute Provider. Added an Identify References virtual attribute provider. These virtual attributes will have values that are the DNs of entries that contain a specified attribute with a value equal to the DN of the entry containing the virtual attribute. For example, this could be used to create a virtual 'directReports' attribute whose values are the DNs of the entries that list the target user as their manager.

  • Self-Service Account Manager (SSAM). SSAM is a web application that provides a user interface for performing common account registration, attribute update and password change tasks against the Datastore, with optional integration with PingFederate and PingAccess products.

Known Issues/Workarounds

The following are known issues in the current version of the PingDirectory Server:

  • When deploying a .war file through the Web Application HTTP Servlet Extension, dependencies bundled in the file may conflict with the server's own dependencies if the server version differs from the version in the .war file. This may cause the Web Application HTTP Servlet Extension or the server itself to not start correctly. For reference, all server dependencies are available in <server root>/lib.

Resolved Issues

The following issues have been resolved with this release of the PingDirectory Server:

Ticket ID Description
DS-1131,DS-14022

Updated the server's password policy support to make a few account usability enhancements:

  • Added support for a get password policy state issues control that can be included in a bind request to obtain information about notices, warnings, and errors that may impact a user's ability to authenticate or interact with the server. The password policy state extended operation has also been updated to provide methods for retrieving this information.
  • Added support for an account activation time as a complement to the existing account expiration time functionality. If an activation time is set in a user's account, the user will not be permitted to authenticate until that time has arrived.
  • Updated the password policy state operation to provide methods for getting, setting, and clearing the user's last login IP address.
DS-1261

The collect-support-data tool now has the option to collect logging information within a specified time range via the '--timeRange' argument.

DS-1706

Updated interactive dsconfig to include an option to toggle between sorting similar properties together or sorting them alphabetically.

DS-3095

Added a new search-logs tool. Similar to the command line tool 'grep,' this tool searches across log files to extract lines matching the provided pattern(s). The search-logs tool can handle multi-line log messages, extract log messages within a given time range, and include rotated log files.

DS-3186

Added the 'listKeysExceedingIndexEntryLimit' argument to the verify-index tool, which enables listing the keys for indexes that have exceeded their index entry limits.

DS-8739

Added the ability to reset user passwords with a single-use, time-limited token that is delivered to the end user through some out-of-band mechanism like SMS or email. After determining the identity of the user for whom the password reset token should be generated, an application can use the new "deliver password reset token" extended operation to cause the server to create and deliver the token to the user. This token can then be provided to the "password modify" extended operation in lieu of the user's current password in order to allow that user to select a new password. Password reset tokens can optionally permit users to reset their passwords even if their account is not usable (for example, because their account is locked or their password is expired).

DS-9842

Added the ability to configure the Globally-Unique Attribute and Unique Attribute plugins with a filter to limit attribute uniqueness checking to a subset of matching entries.

DS-10010

Reduced the memory overhead of debug logging in high throughput environments by sharing logging buffers across multiple threads.

DS-10283

Custom HTTP loggers are no longer permitted to modify the requests and responsesbeing logged. Calling a forbidden method will result in a subclass of UnsupportedOperationException. For requests, the forbidden methods are authenticate, getReader, login, logout and setCharacterEncoding. For responses, the forbidden methods are addCookie, addHeader, addIntHeader, flushBuffer, getOutputStream, getWriter, reset, sendError, sendRedirect, setBufferSize, setCharacterEncoding, setContentLength, setContentType, setHeader, setIntHeader, setLocale and setStatus.

DS-10775

Added a new Commonly-Used Passwords instance of the dictionary password validator that uses a dictionary file with 10,000 of the most common user passwords as determined by analysis of data from a number of security breaches. Because these passwords are so popular among end users, they are also very commonly guessed by attackers trying to compromise end user accounts.

The Commonly-Used Passwords validator is defined in the out-of-the-box configuration, but is only invoked by the Secure Password Policy by default.

DS-10843

Added support for a "name with entryUUID" request control. If this control is included in an add request, the entry will be added with a distinguished name whose RDN contains only the entryUUID attribute. This offers a number of potential benefits:

  • It can help preserve data privacy by ensuring the entry DN does not include sensitive or personally-identifying information.
  • It can reduce the need for modify DN operations, since entries are not named with user attributes that have the potential to change.
  • It can serve as a convenience for entries in which there is no obvious, guaranteed-unique attribute (or combination of attributes) to use for naming those entries.
DS-11067

Added properties to the task backend for limiting the number of log messages retained in task entries, in order to limit the size of the in-memory representation of those entries. All log messages generated by a task will still be recorded in the server error log, even if they are not all retained in the corresponding entry in the task backend.

DS-11522

Updated the server's JVM arguments to always log garbage collection information to a rotating set of log files stored within logs/jvm/gc.log.N. The file system usage is limited to 300MB. If the server had previously been configured with VERBOSE_GC, then garbage collection logging information will no longer be logged to logs/server.out.

DS-11823,DS-13535,DS-13894

Deprecated the invalid-attribute-syntax-behavior global configuration property in favor of a new permit-syntax-violations-for-attribute global configuration property. The new option makes it possible to allow malformed values for an explicitly-specified set of attribute types, whereas the former option could only be used to enable or disable syntax enforcement for all attribute types.

When migrating from a directory service that did not properly enforce attribute syntax compliance, it is strongly recommended that the data be cleaned to correct any malformed values that it may contain. However, in cases where that may not be immediately feasible, it is strongly recommended that syntax validation be relaxed only for attribute types that are known to have problems so that it will still be performed for other attribute types to prevent inadvertently introducing additional malformed values.

In the event that an LDIF file contains malformed values, the import-ldif tool will now provide a list of the attribute types with attribute syntax violations and the number of malformed values identified for each attribute type. As before, the specific violations can be identified by instructing the import-ldif tool to generate a rejects file, which will include a comment with each rejected entry to describe the reason the entry was rejected.

In addition, the server will now always perform syntax validation for the aci attribute type, regardless of the values of the invalid-attribute-syntax-behavior and permit-syntax-violations-for-attribute properties. This will provide additional assurance that malformed access control instructions cannot be introduced into the server during LDIF import processing. The server will still discover and validate all ACIs on startup, and will still place itself in lockdown mode on finding a malformed ACI rather than attempting to run with an incomplete access control configuration.

DS-12106

Added support for a new "Haystack" password validator based on the concept of password haystacks as described at https://www.grc.com/haystack.htm. It estimates the strength of a password using a combination of its length and the types of characters that it contains (e.g., a longer password containing only lowercase letters may be stronger than a shorter password containing a mix of uppercase and lowercase letters, numbers, and symbols).

The Haystack password validator is defined in the out-of-the-box configuration but is only enabled by default in the secure password policy.

DS-12107,DS-12137

Added features to allow clients to better determine the set of requirements that the server will impose for user passwords. The get password quality requirements extended operation can be used to retrieve information about the requirements before an attempted password change. Those requirements can be conveyed to the end user, and can potentially be used to enable some types of client-side validation to identify problems with a password before it is sent to the server. The password validation details request control can be included in an add request, a modify request, or a password modify extended request to identify which specific validation requirements may not have been met by the password provided in the request.

Password validators can be configured with user-friendly messages that better describe the constraints that the validator will impose for passwords, and that the validator should return if a proposed password does not satisfy those constraints. The server will generate these messages if they are not provided in the configuration.

DS-12123

Updated the Configuration API output where properties and their values are listed to include those that are undefined.

DS-12138

Added support for a JSON object attribute syntax, which can be used for attribute types whose values are JSON objects. The syntax requires that each value of this type is a valid JSON object. Two matching rules have also been added for use in conjunction with the JSON object syntax: jsonObjectExactMatch and jsonObjectFilterExtensibleMatch.

The jsonObjectExactMatch equality matching rule is used in evaluating equality filters in search operations, as well as for matching performed against JSON object attributes for add, compare, and modify operations. It determines whether two values are logically-equivalent JSON objects. The field names used in both objects must match exactly (although fields may appear in different orders). The values of each field must have the same data types. String values will be compared in a case-insensitive manner. The order of elements in arrays will be considered significant.

The jsonObjectFilterExtensibleMatch matching rule can perform more powerful matching against JSON objects. The assertion values for these extensible matching filters should be JSON objects that express the constraints for the matching. These JSON object filters are described in detail in the Javadoc documentation (available in the Commercial Edition of the UnboundID LDAP SDK for Java) for the com.unboundid.ldap.sdk.unboundidds.json.JSONObjectFilter class and its subclasses. Although the LDAP SDK can facilitate searches with this matching rule, these searches can be issued through any LDAP client API that supports extensible matching.

Indexing is supported only for the jsonObjectExactMatch matching rule. If possible, non-baseObject searches that use the jsonObjectFilterExtensibleMatch matching rule should be wrapped in an LDAP AND filter that also contains one or more indexed components so that the search can be processed more efficiently.

DS-12139,DS-12917,DS-13476,DS-13538

Enhanced the server's support for storing and interacting with JSON objects.

It is now possible to configure indexes for specified fields inside JSON objects to accelerate JSON object filter extensible match search operations. Indexes can be used for fields with boolean, integer, null, and string values and JSON object filters of type equals, equalsAny, greaterThan, lessThan, and substring, as well as AND filters that contain at least one indexed component and OR filters that contain only indexed components. Note that greaterThan and lessThan filters that target string values can only be indexed if they use case-insensitive matching.

It is now possible to indicate that the values of specified fields (ideally fields with a relatively small set of distinct values) should be tokenized when they are stored in the database. Tokenized values can be stored more efficiently, and consume less space in memory and on disk.

It is now possible to define a number of constraints for the fields that may be included in JSON objects stored in values of a specified attribute type. Constraints that may be imposed on a JSON field include: - Require values of the field to have a specified data type. - Indicate whether the field is required or optional. - Indicate whether the field is permitted to have multiple values in an array. If a field is permitted to have array values, then it is also possible to place restrictions on the number of elements that may be present in the array. - Indicate whether the field is permitted to have a value that is the null primitive as an alternative to values of the indicated data type. - Restrict values of string fields to a predefined set of allowed values, to values matching a given regular expression, or to values of a specified length. - Restrict values of numeric fields with upper and lower bounds.

DS-12178

A new index is now considered trusted if the server can determine that the associated attribute type (or JSON field for a JSON index) is not used in the data already contained in the target backend. If an index is automatically trusted, it is not necessary to use the rebuild-index tool to initialize that index.

DS-12182

The setup tool has been updated to use HTTPS for initial configuration. Unsecure HTTP can be enabled post-setup, or by using non-interactive setup.

DS-12218

Updated the server to automatically monitor and report the length of time each operation spends waiting in the work queue before a worker thread can begin to process it.

DS-12245

The Configuration API has been updated to support filtering, sorting, and paging for object list operations. See the Administration guide for usage.

DS-12263

Updated the local DB backend so that changes to the db-checkpointer-wakeup-interval property no longer require a restart to take effect, and to expose new monitor attributes with useful information about the processing performed by the database cleaner.

DS-12285

Fixed an issue where changes to SMTP External Server configurations did not take effect until after a server restart.

DS-12287

Addressed cases where some messages may be suppressed in logs and alerts.

DS-12312

Updated the report generated by import-ldif to include database cache requirements for each possible cache-mode per backend database (e.g., attribute index). This aids tuning environments that cannot be fully cached.

DS-12313

Changed the default password policy behavior to prevent users from changing their passwords to their current password value. This logic will apply regardless of password history settings.

DS-12318

Added a configuration option to enable a wait period before removing a 'server unavailable' alert after a garbage collection task is performed. This allows sub-systems like replication to restart before the server becomes available again. For the Periodic GC Plugin, this option is 'delay-post-gc.' For a Forced GC Task entry, the attribute is named 'ds-task-delay-post-gc.' Both options take a value in milliseconds, and have a default value equivalent to 20 seconds.

DS-12319

Updated UnboundID work queue processing to log expensive work queue operations and diagnostic thread stack traces when a queue backlog alarm is raised.

DS-12443

Fixed an issue that generated the following error message, but did not impact the current operation: "An unexpected error occurred while notifying a change notification listener of a modify operation: RuntimeException: The specified condition must be true. The error occurred at com.unboundid.directory.server.types.AuthenticationInfo.replaceUserEntries."

DS-12483

Added support for running on Oracle Java 8 and OpenJDK 8 platforms.

DS-12496

Added logging of all HTTP requests disallowed due to CORS. This should make it easier to debug HTTP 403/Forbidden errors.

DS-12560,DS-12561

Updated the server to avoid the use of the server-side sort and virtual list view request controls in search requests that span multiple subtree views or multiple entry-balanced backend sets. If the server cannot honor a non-critical server-side sort or virtual list view control, then it will process the search operation as if the control had not been included in the request. If the server cannot honor a critical server-side sort or virtual list view control, then it will return an error result to the client.

DS-12576

Update the Detailed HTTP Operation Log Publisher to log the correct return code (404 NOT FOUND) when a request is not handled by defined endpoints.

DS-12579

The server can now detect an "out of file handles" situation on the operating system, and shut down to prevent running in an unreliable state.

DS-12594,DS-12596

Added support for three new extended operations for interacting with single-use tokens:

  • The "get supported OTP delivery mechanisms" operation provides information about which one-time password delivery mechanisms are configured in the server, and which of those are available for a specified user.
  • The "deliver single-use token" operation can generate a token value and provide it to a specified user through an out-of-band communication mechanism like email, SMS, or voice call.
  • The "consume single-use token" operation indicates that the user has received a single-use token from the "deliver single-use token" operation, and to consume that token so that it cannot be reused.
DS-12610

Fixed an issue where configuring numeric IPv4 address filtering by connection criteria in a log publisher performed unnecessary reverse host name lookups.

DS-12618

Updated the notification destination cn=monitor entry (objectclass of ds-notification-destination-monitor-entry) to include an attribute, ds-notification-age-of-next-pending-change-seconds, which tracks how out-of-date the destination is in seconds. Values are only maintained on the master server for that domain (ds-notification-master=true). A value of 0 on the master server for that domain indicates that the destination is up-to-date. This attribute can be used in a gauge to generate alarms if a destination gets too far behind.

DS-12627

Updated the LDAP connection handler to enable the use of multiple threads for accepting connections and preparing them for use. This improves concurrency for deployments in which the process of accepting a new connection may take some time to complete, possibly because of expensive DNS lookups or invoking time-consuming post-connect plugins).

DS-12681,DS-13475

Improved the server's support for selecting TLS cipher suites. When the server is configured to use a specific set of cipher suites, it will now always validate that all of the configured suites are supported by the JVM. When the server is not configured to use a specific set of cipher suites, it will now customize the set of default suites to prioritize those using strong cryptography (especially those that offer forward secrecy), and exclude suites with known weaknesses.

DS-12700

JDBC external servers now give precedence to settings in the jdbc-driver-url property, over other dsconfig JDBC Managed Object settings for host-name, port, and database-name. The jdbc-driver-url property setting can be used instead of the other JDBC Managed Object settings.

DS-12727

Added a gauge to monitor the number of available file descriptors. This Available File Descriptors gauge can detect if a server if running out of file descriptors and degrade the server appropriately.

DS-12798

MakeLDIF templates now have the ability to escape special characters curly braces, angle brackets, and square brackets using a backslash. See config/MakeLDIF/examples-of-all-tags.template for further examples.

DS-12799

The following UnboundID product names have changed: - Identity Datastore to Datastore - Identity Proxy to Proxy Server - Identity Data Sync Server to Data Sync Server - Identity Broker to Data Broker

DS-12833

Updated the alert handler configuration to indicate whether the alert handler should be invoked asynchronously in a background thread rather than by the thread that generated the alert. For alerts generated during the course of processing an operation, invoking potentially time-consuming alert handlers in a background thread can avoid adversely impacting the response time for that operation while still ensuring that administrators are made aware of the issue that arose.

DS-12833

Updated the server to provide support for SMTP connection pooling. When sending an email message, the server will attempt to reuse an existing SMTP connection rather than establishing a new connection for each message.

DS-12833

Updated the account status notification handler configuration to indicate whether the handler should be invoked asynchronously in a background thread rather than by the thread that triggered the notification. For account status notifications generated during the course of processing an operation, invoking notification handlers in a background thread can avoid adversely impacting the response time for that operation.

DS-12880

Fixed a rare condition that might cause the logger rotation and retention thread to exit under heavy file system load or a network file system outage.

DS-12897

Fixed a bug where using the advanced arguments of some tools would result in changing the saved complexity settings for the dsconfig tool.

DS-12909

Fixed the Local DB Backend configuration help text for deadlock-retry-limit, which incorrectly stated that a value of zero would result in unlimited retry attempts. That value actually results in no retry attempts.

DS-12933

Updated the server to reject search requests that attempt to make use of an invalid JSON object filter. The server would previously return a success result with no matching entries.

DS-12943

Fixed an issue where the Datastore parsed the last logon time value using the wrong time zone. The incorrect time affected password policy decisions and was delivered in the response to a password policy state extended request.

DS-12969

Improved server locking used by dsconfig in offline batch mode, so that the server lock is held for the entire batch duration, instead of for each invocation. Also, reduced the probability of contention for file locks used by server tools to determine the server status.

DS-12982

Updated the installer to increase the maximum suggested JVM size on Linux systems with at least 48 GB of RAM.

DS-13124

Fixed an issue where debug logging at a fine-level could consume large amounts of memory.

DS-13140

Fixed a problem that could cause the server to incorrectly handle the require-change-by-time property in the password policy configuration.

DS-13163

Addressed an issue where data definition language (DDL) log field mappings for the JDBC error log were not previously documented.

DS-13182

Updated the local DB backend to always try to keep exploded indexes in the database cache, and to always load exploded indexes into the cache on startup if database preloading is enabled, even if the database containing non-exploded index data for the associated index is configured with a lesser cache mode or preload configuration. Because reading from exploded indexes requires much more database interaction than non-exploded indexes, this can dramatically improve the performance of exploded index accesses for deployments in which it is not possible to fully cache all data in the backend.

DS-13206

Updated the server to use the latest 6.3.8 release of the Berkeley DB Java Edition.

DS-13207

Improved the server shutdown time in situations when there is a database cleaner backlog. This also ensures a faster start up time since database recovery isn't needed.

DS-13426

Updated the server to use the latest 6.3.9 release of the Berkeley DB Java Edition.

DS-13453

Fixed a defect where a deny ACI with target attrs would prevent Modify DN operations from succeeding, even when the target attrs did not include any RDN attrs.

DS-13481

Fixed an edge case problem in which a password change could result in the previous password not being included in the password history.

DS-13484

Fixed a conflict between the Changelog Password Encryption plugin and the replication historical ds-sync-hist attribute that would result in a "historical information for this attribute seems to be corrupt" error message in the logs.

DS-13552

Fixed a log publisher defect that would result in an unreadable file when both compression and signing were enabled at the same time.

DS-13554

Updated the server to allow an initial heap size over 128 GB. Due to limitations of older JVMs, this was previously capped at 128 GB, even when the maximum heap size was larger.

DS-13559

Fixed an issue that could cause the server to overlook attribute options in an indexed search filter.

DS-13644

Fixed an issue with the Pass Through Authentication Plugin where if a DN map was not used, the controls from the native bind operation were included in the pass through authentication request.

DS-13678

Updated the create-systemd-script tool by adding resource limits for available open file descriptors (NOFILE), and shared memory reservations (MEMLOCK). The generated script lists the recommended file descriptors limit and the resource limit setting for enabling large page support. The settings in the create-systemd-script output supersedes prior documentation for setting the number of open file descriptors on non-systemd systems.

DS-13727

Added an Identify References virtual attribute provider. Virtual attributes of this type will have values that are the DNs of entries that contain a specified attribute with a value that matches the DN of the entry containing the virtual attribute. For example, this could be used to create a virtual directReports attribute whose values are the DNs of the entries that list the target user as their manager.

DS-13748

Updated the ldif-diff tool to ensure that change records for delete operations will be ordered to ensure that a delete for a parent entry will never come before the deletes for its children.

DS-13771

Updated the initial output of export-ldif to report that it is calculating a disk-ordered cursor rather than "Exported 0 entries."

DS-13783

Updated the server to better utilize worker threads and reduce the potential for a work queue backlog when processing multiple concurrent long-running operations.

DS-13794

Updated the server to generate an administrative alert if it detects that a database environment was not closed cleanly and may require a time-consuming recovery process.

DS-13820

Fixed an issue involving transactions sent through a Proxy Server with Entry Balancing configured. If the transaction contained requests that targeted entries that were not in the global index, then duplicate requests were included in the resulting Multi-Update operation forwarded to the Datastore.

DS-13856

Updated the server to use the latest 6.4.9 release of Berkeley DB Java Edition.

DS-13862

Updated the server to discourage conflicts between indexes and virtual attributes. A search that targets an indexed attribute will only identify entries with real values as potential matches, and may omit entries that match the filter if they match because of a virtually-generated value.

DS-13899

Fixed an issue with large password history duration values that may have prevented some passwords in the history from being considered a match when a new password with the same value was added.

DS-13983

Fixed an issue where dynamic group membership searches lead to resource and memory leaks.

DS-14014

Updated the server to use the latest 6.4.12 release of the Berkeley DB Java Edition. This version addresses a possible data corruption bug in versions 6.3.0 to 6.4.11.

DS-14036

Fixed an issue where password history values could be duplicated in replicated environments.

DS-14060

Fix an issue in the SCIM interface where an attribute required by the SCIM schema could be deleted by a PATCH operation.

DS-14074

Added the ability to protect Velocity templates using the basic authentication scheme.

DS-14099

Fixed an issue where replication would stall while performing searches on LDAP referrals.

DS-14116

Fixed issues pertaining to search references. One caused replication to stall, and the other caused multiple copies of the same reference to be returned in a single search.

DS-14133

Fixed a couple of corner cases in which the server could treat smart referrals improperly. This includes search operations based above smart referrals contained in entries whose DNs contain escaped commas, as well as some search and compare operations based at least one level below a smart referral.

DS-14140

The ldifsearch command now supports the option "---isCompressed" for LDIF files that have been compressed with gzip.

DS-14259

Updated the email OTP delivery mechanism to allow retrieving email addresses from fields contained inside JSON objects, optionally using a JSON object filter to select which of several addresses should be used (e.g., only attempt to use verified email addresses). Similar changes have been made to the Twilio OTP delivery mechanism for obtaining phone numbers for SMS messages.

DS-14311

Fixed a race condition that could arise from simultaneous attempts to add entries with an attribute value that would cause an exploded index key to exceed its index entry limit.

DS-14349

Fixed an issue with the collect-support-data tool when using the --pid argument. Only one jstack was being collected, instead of using the amount specified by the --maxJstacks argument.

DS-14398,DS-14399

Added password storage schemes that leverage the Bcrypt and scrypt key derivation functions. These storage schemes require the free and open source Bouncy Castle library, which is not included with the server. This library must be obtained from https://bouncycastle.org/ and placed in the server lib directory before these storage schemes can be used.