What's New
These are new features for this release of the PingDirectory Server:
- Improvements to Datastore’s Native JSON Attribute Support - The previous release of the Datastore included native support for compactly storing JSON objects in LDAP attributes, and for evaluating filters to match on fields inside those JSON objects. This release adds support for enforcing constraints on the fields that may be included in JSON objects, for indexing field values for improved search performance and flexibility, and for tokenizing commonly-used string values to further reduce the data footprint.
- New Pasword Validators - The Datastore now includes the Haystacks Password Validator, based on the Gibson Research Corporation Password Haystacks concept, and the Commonly-Used Passwords Dictionary Validator, which ensures that a proposed password is not one of 10,000 commonly used passwords.
- Can Request an entryUUID rdn Value when Adding an Entry - When adding a new entry to the server, the client can now request that the server-generated entryUUID be used as the RDN attribute for the entry. This improves privacy by ensuring that the entry DN will not include any personally-identifying information, and it is a convenience for application developers by eliminating the potential need for modify DN operations.
- Implemented a Virtual Attribute Provider - Added an Identify References virtual attribute provider. These virtual attributes will have values that are the DNs of entries that contain a specified attribute with a value equal to the DN of the entry containing the virtual attribute. For example, this could be used to create a virtual 'directReports' attribute whose values are the DNs of the entries that list the target user as their manager.
- Self-Service Account Manager (SSAM) - SSAM is a web application that provides a user interface for performing common account registration, attribute update and password change tasks against the Datastore, with optional integration with PingFederate and PingAccess products.
Known Issues/Workarounds
The following are known issues in the current version of the PingDirectory Server:
- When deploying a .war file through the Web Application HTTP Servlet Extension, dependencies bundled in the file may conflict with the server's own dependencies if the server version differs from the version in the .war file. This may cause the Web Application HTTP Servlet Extension or the server itself to not start correctly. For reference, all server dependencies are available in <server root>/lib.
Resolved Issues
The following issues have been resolved with this release of the PingDirectory Server:
Ticket ID | Description |
---|---|
DS-1131,DS-14022 |
Updated the server's password policy support to make a few account usability enhancements:
|
DS-1261 |
The collect-support-data tool now has the option to collect logging information within a specified time range via the '--timeRange' argument. |
DS-1706 |
Updated interactive dsconfig to include an option to toggle between sorting similar properties together or sorting them alphabetically. |
DS-3095 |
Added a new search-logs tool. Similar to the command line tool 'grep,' this tool searches across log files to extract lines matching the provided pattern(s). The search-logs tool can handle multi-line log messages, extract log messages within a given time range, and include rotated log files. |
DS-3186 |
Added the 'listKeysExceedingIndexEntryLimit' argument to the verify-index tool, which enables listing the keys for indexes that have exceeded their index entry limits. |
DS-8739 |
Added the ability to reset user passwords with a single-use, time-limited token that is delivered to the end user through some out-of-band mechanism like SMS or email. After determining the identity of the user for whom the password reset token should be generated, an application can use the new "deliver password reset token" extended operation to cause the server to create and deliver the token to the user. This token can then be provided to the "password modify" extended operation in lieu of the user's current password in order to allow that user to select a new password. Password reset tokens can optionally permit users to reset their passwords even if their account is not usable (for example, because their account is locked or their password is expired). |
DS-9842 |
Added the ability to configure the Globally-Unique Attribute and Unique Attribute plugins with a filter to limit attribute uniqueness checking to a subset of matching entries. |
DS-10010 |
Reduced the memory overhead of debug logging in high throughput environments by sharing logging buffers across multiple threads. |
DS-10283 |
Custom HTTP loggers are no longer permitted to modify the requests and responsesbeing logged. Calling a forbidden method will result in a subclass of UnsupportedOperationException. For requests, the forbidden methods are authenticate, getReader, login, logout and setCharacterEncoding. For responses, the forbidden methods are addCookie, addHeader, addIntHeader, flushBuffer, getOutputStream, getWriter, reset, sendError, sendRedirect, setBufferSize, setCharacterEncoding, setContentLength, setContentType, setHeader, setIntHeader, setLocale and setStatus. |
DS-10775 |
Added a new Commonly-Used Passwords instance of the dictionary password validator that uses a dictionary file with 10,000 of the most common user passwords as determined by analysis of data from a number of security breaches. Because these passwords are so popular among end users, they are also very commonly guessed by attackers trying to compromise end user accounts. The Commonly-Used Passwords validator is defined in the out-of-the-box configuration, but is only invoked by the Secure Password Policy by default. |
DS-10843 |
Added support for a "name with entryUUID" request control. If this control is included in an add request, the entry will be added with a distinguished name whose RDN contains only the entryUUID attribute. This offers a number of potential benefits:
|
DS-11067 |
Added properties to the task backend for limiting the number of log messages retained in task entries, in order to limit the size of the in-memory representation of those entries. All log messages generated by a task will still be recorded in the server error log, even if they are not all retained in the corresponding entry in the task backend. |
DS-11522 |
Updated the server's JVM arguments to always log garbage collection information to a rotating set of log files stored within logs/jvm/gc.log.N. The file system usage is limited to 300MB. If the server had previously been configured with VERBOSE_GC, then garbage collection logging information will no longer be logged to logs/server.out. |
DS-11823,DS-13535,DS-13894 |
Deprecated the invalid-attribute-syntax-behavior global configuration property in favor of a new permit-syntax-violations-for-attribute global configuration property. The new option makes it possible to allow malformed values for an explicitly-specified set of attribute types, whereas the former option could only be used to enable or disable syntax enforcement for all attribute types. When migrating from a directory service that did not properly enforce attribute syntax compliance, it is strongly recommended that the data be cleaned to correct any malformed values that it may contain. However, in cases where that may not be immediately feasible, it is strongly recommended that syntax validation be relaxed only for attribute types that are known to have problems so that it will still be performed for other attribute types to prevent inadvertently introducing additional malformed values. In the event that an LDIF file contains malformed values, the import-ldif tool will now provide a list of the attribute types with attribute syntax violations and the number of malformed values identified for each attribute type. As before, the specific violations can be identified by instructing the import-ldif tool to generate a rejects file, which will include a comment with each rejected entry to describe the reason the entry was rejected. In addition, the server will now always perform syntax validation for the aci attribute type, regardless of the values of the invalid-attribute-syntax-behavior and permit-syntax-violations-for-attribute properties. This will provide additional assurance that malformed access control instructions cannot be introduced into the server during LDIF import processing. The server will still discover and validate all ACIs on startup, and will still place itself in lockdown mode on finding a malformed ACI rather than attempting to run with an incomplete access control configuration. |
DS-12106 |
Added support for a new "Haystack" password validator based on the concept of password haystacks as described at https://www.grc.com/haystack.htm. It estimates the strength of a password using a combination of its length and the types of characters that it contains (e.g., a longer password containing only lowercase letters may be stronger than a shorter password containing a mix of uppercase and lowercase letters, numbers, and symbols). The Haystack password validator is defined in the out-of-the-box configuration but is only enabled by default in the secure password policy. |
DS-12107,DS-12137 |
Added features to allow clients to better determine the set of requirements that the server will impose for user passwords. The get password quality requirements extended operation can be used to retrieve information about the requirements before an attempted password change. Those requirements can be conveyed to the end user, and can potentially be used to enable some types of client-side validation to identify problems with a password before it is sent to the server. The password validation details request control can be included in an add request, a modify request, or a password modify extended request to identify which specific validation requirements may not have been met by the password provided in the request. Password validators can be configured with user-friendly messages that better describe the constraints that the validator will impose for passwords, and that the validator should return if a proposed password does not satisfy those constraints. The server will generate these messages if they are not provided in the configuration. |
DS-12123 |
Updated the Configuration API output where properties and their values are listed to include those that are undefined. |
DS-12138 |
Added support for a JSON object attribute syntax, which can be used for attribute types whose values are JSON objects. The syntax requires that each value of this type is a valid JSON object. Two matching rules have also been added for use in conjunction with the JSON object syntax: jsonObjectExactMatch and jsonObjectFilterExtensibleMatch. The jsonObjectExactMatch equality matching rule is used in evaluating equality filters in search operations, as well as for matching performed against JSON object attributes for add, compare, and modify operations. It determines whether two values are logically-equivalent JSON objects. The field names used in both objects must match exactly (although fields may appear in different orders). The values of each field must have the same data types. String values will be compared in a case-insensitive manner. The order of elements in arrays will be considered significant. The jsonObjectFilterExtensibleMatch matching rule can perform more powerful matching against JSON objects. The assertion values for these extensible matching filters should be JSON objects that express the constraints for the matching. These JSON object filters are described in detail in the Javadoc documentation (available in the Commercial Edition of the UnboundID LDAP SDK for Java) for the com.unboundid.ldap.sdk.unboundidds.json.JSONObjectFilter class and its subclasses. Although the LDAP SDK can facilitate searches with this matching rule, these searches can be issued through any LDAP client API that supports extensible matching. Indexing is supported only for the jsonObjectExactMatch matching rule. If possible, non-baseObject searches that use the jsonObjectFilterExtensibleMatch matching rule should be wrapped in an LDAP AND filter that also contains one or more indexed components so that the search can be processed more efficiently. |
DS-12139,DS-12917,DS-13476,DS-13538 |
Enhanced the server's support for storing and interacting with JSON objects. It is now possible to configure indexes for specified fields inside JSON objects to accelerate JSON object filter extensible match search operations. Indexes can be used for fields with boolean, integer, null, and string values and JSON object filters of type equals, equalsAny, greaterThan, lessThan, and substring, as well as AND filters that contain at least one indexed component and OR filters that contain only indexed components. Note that greaterThan and lessThan filters that target string values can only be indexed if they use case-insensitive matching. It is now possible to indicate that the values of specified fields (ideally fields with a relatively small set of distinct values) should be tokenized when they are stored in the database. Tokenized values can be stored more efficiently, and consume less space in memory and on disk. It is now possible to define a number of constraints for the fields that may be included in JSON objects stored in values of a specified attribute type. Constraints that may be imposed on a JSON field include: - Require values of the field to have a specified data type. - Indicate whether the field is required or optional. - Indicate whether the field is permitted to have multiple values in an array. If a field is permitted to have array values, then it is also possible to place restrictions on the number of elements that may be present in the array. - Indicate whether the field is permitted to have a value that is the null primitive as an alternative to values of the indicated data type. - Restrict values of string fields to a predefined set of allowed values, to values matching a given regular expression, or to values of a specified length. - Restrict values of numeric fields with upper and lower bounds. |
DS-12178 |
A new index is now considered trusted if the server can determine that the associated attribute type (or JSON field for a JSON index) is not used in the data already contained in the target backend. If an index is automatically trusted, it is not necessary to use the rebuild-index tool to initialize that index. |
DS-12182 |
The setup tool has been updated to use HTTPS for initial configuration. Unsecure HTTP can be enabled post-setup, or by using non-interactive setup. |
DS-12218 |
Updated the server to automatically monitor and report the length of time each operation spends waiting in the work queue before a worker thread can begin to process it. |
DS-12245 |
The Configuration API has been updated to support filtering, sorting, and paging for object list operations. See the Administration guide for usage. |
DS-12263 |
Updated the local DB backend so that changes to the db-checkpointer-wakeup-interval property no longer require a restart to take effect, and to expose new monitor attributes with useful information about the processing performed by the database cleaner. |
DS-12285 |
Fixed an issue where changes to SMTP External Server configurations did not take effect until after a server restart. |
DS-12287 |
Addressed cases where some messages may be suppressed in logs and alerts. |
DS-12312 |
Updated the report generated by import-ldif to include database cache requirements for each possible cache-mode per backend database (e.g., attribute index). This aids tuning environments that cannot be fully cached. |
DS-12313 |
Changed the default password policy behavior to prevent users from changing their passwords to their current password value. This logic will apply regardless of password history settings. |
DS-12318 |
Added a configuration option to enable a wait period before removing a 'server unavailable' alert after a garbage collection task is performed. This allows sub-systems like replication to restart before the server becomes available again. For the Periodic GC Plugin, this option is 'delay-post-gc.' For a Forced GC Task entry, the attribute is named 'ds-task-delay-post-gc.' Both options take a value in milliseconds, and have a default value equivalent to 20 seconds. |
DS-12319 |
Updated UnboundID work queue processing to log expensive work queue operations and diagnostic thread stack traces when a queue backlog alarm is raised. |
DS-12443 |
Fixed an issue that generated the following error message, but did not impact the current operation: "An unexpected error occurred while notifying a change notification listener of a modify operation: RuntimeException: The specified condition must be true. The error occurred at com.unboundid.directory.server.types.AuthenticationInfo.replaceUserEntries." |
DS-12483 |
Added support for running on Oracle Java 8 and OpenJDK 8 platforms. |
DS-12496 |
Added logging of all HTTP requests disallowed due to CORS. This should make it easier to debug HTTP 403/Forbidden errors. |
DS-12560,DS-12561 |
Updated the server to avoid the use of the server-side sort and virtual list view request controls in search requests that span multiple subtree views or multiple entry-balanced backend sets. If the server cannot honor a non-critical server-side sort or virtual list view control, then it will process the search operation as if the control had not been included in the request. If the server cannot honor a critical server-side sort or virtual list view control, then it will return an error result to the client. |
DS-12576 |
Update the Detailed HTTP Operation Log Publisher to log the correct return code (404 NOT FOUND) when a request is not handled by defined endpoints. |
DS-12579 |
The server can now detect an "out of file handles" situation on the operating system, and shut down to prevent running in an unreliable state. |
DS-12594,DS-12596 |
Added support for three new extended operations for interacting with single-use tokens:
|
DS-12610 |
Fixed an issue where configuring numeric IPv4 address filtering by connection criteria in a log publisher performed unnecessary reverse host name lookups. |
DS-12618 |
Updated the notification destination cn=monitor entry (objectclass of ds-notification-destination-monitor-entry) to include an attribute, ds-notification-age-of-next-pending-change-seconds, which tracks how out-of-date the destination is in seconds. Values are only maintained on the master server for that domain (ds-notification-master=true). A value of 0 on the master server for that domain indicates that the destination is up-to-date. This attribute can be used in a gauge to generate alarms if a destination gets too far behind. |
DS-12627 |
Updated the LDAP connection handler to enable the use of multiple threads for accepting connections and preparing them for use. This improves concurrency for deployments in which the process of accepting a new connection may take some time to complete, possibly because of expensive DNS lookups or invoking time-consuming post-connect plugins). |
DS-12681,DS-13475 |
Improved the server's support for selecting TLS cipher suites. When the server is configured to use a specific set of cipher suites, it will now always validate that all of the configured suites are supported by the JVM. When the server is not configured to use a specific set of cipher suites, it will now customize the set of default suites to prioritize those using strong cryptography (especially those that offer forward secrecy), and exclude suites with known weaknesses. |
DS-12700 |
JDBC external servers now give precedence to settings in the jdbc-driver-url property, over other dsconfig JDBC Managed Object settings for host-name, port, and database-name. The jdbc-driver-url property setting can be used instead of the other JDBC Managed Object settings. |
DS-12727 |
Added a gauge to monitor the number of available file descriptors. This Available File Descriptors gauge can detect if a server if running out of file descriptors and degrade the server appropriately. |
DS-12798 |
MakeLDIF templates now have the ability to escape special characters curly braces, angle brackets, and square brackets using a backslash. See config/MakeLDIF/examples-of-all-tags.template for further examples. |
DS-12799 |
The following UnboundID product names have changed: - Identity Datastore to Datastore - Identity Proxy to Proxy Server - Identity Data Sync Server to Data Sync Server - Identity Broker to Data Broker |
DS-12833 |
Updated the alert handler configuration to indicate whether the alert handler should be invoked asynchronously in a background thread rather than by the thread that generated the alert. For alerts generated during the course of processing an operation, invoking potentially time-consuming alert handlers in a background thread can avoid adversely impacting the response time for that operation while still ensuring that administrators are made aware of the issue that arose. |
DS-12833 |
Updated the server to provide support for SMTP connection pooling. When sending an email message, the server will attempt to reuse an existing SMTP connection rather than establishing a new connection for each message. |
DS-12833 |
Updated the account status notification handler configuration to indicate whether the handler should be invoked asynchronously in a background thread rather than by the thread that triggered the notification. For account status notifications generated during the course of processing an operation, invoking notification handlers in a background thread can avoid adversely impacting the response time for that operation. |
DS-12880 |
Fixed a rare condition that might cause the logger rotation and retention thread to exit under heavy file system load or a network file system outage. |
DS-12897 |
Fixed a bug where using the advanced arguments of some tools would result in changing the saved complexity settings for the dsconfig tool. |
DS-12909 |
Fixed the Local DB Backend configuration help text for deadlock-retry-limit, which incorrectly stated that a value of zero would result in unlimited retry attempts. That value actually results in no retry attempts. |
DS-12933 |
Updated the server to reject search requests that attempt to make use of an invalid JSON object filter. The server would previously return a success result with no matching entries. |
DS-12943 |
Fixed an issue where the Datastore parsed the last logon time value using the wrong time zone. The incorrect time affected password policy decisions and was delivered in the response to a password policy state extended request. |
DS-12969 |
Improved server locking used by dsconfig in offline batch mode, so that the server lock is held for the entire batch duration, instead of for each invocation. Also, reduced the probability of contention for file locks used by server tools to determine the server status. |
DS-12982 |
Updated the installer to increase the maximum suggested JVM size on Linux systems with at least 48 GB of RAM. |
DS-13124 |
Fixed an issue where debug logging at a fine-level could consume large amounts of memory. |
DS-13140 |
Fixed a problem that could cause the server to incorrectly handle the require-change-by-time property in the password policy configuration. |
DS-13163 |
Addressed an issue where data definition language (DDL) log field mappings for the JDBC error log were not previously documented. |
DS-13182 |
Updated the local DB backend to always try to keep exploded indexes in the database cache, and to always load exploded indexes into the cache on startup if database preloading is enabled, even if the database containing non-exploded index data for the associated index is configured with a lesser cache mode or preload configuration. Because reading from exploded indexes requires much more database interaction than non-exploded indexes, this can dramatically improve the performance of exploded index accesses for deployments in which it is not possible to fully cache all data in the backend. |
DS-13206 |
Updated the server to use the latest 6.3.8 release of the Berkeley DB Java Edition. |
DS-13207 |
Improved the server shutdown time in situations when there is a database cleaner backlog. This also ensures a faster start up time since database recovery isn't needed. |
DS-13426 |
Updated the server to use the latest 6.3.9 release of the Berkeley DB Java Edition. |
DS-13453 |
Fixed a defect where a deny ACI with target attrs would prevent Modify DN operations from succeeding, even when the target attrs did not include any RDN attrs. |
DS-13481 |
Fixed an edge case problem in which a password change could result in the previous password not being included in the password history. |
DS-13484 |
Fixed a conflict between the Changelog Password Encryption plugin and the replication historical ds-sync-hist attribute that would result in a "historical information for this attribute seems to be corrupt" error message in the logs. |
DS-13552 |
Fixed a log publisher defect that would result in an unreadable file when both compression and signing were enabled at the same time. |
DS-13554 |
Updated the server to allow an initial heap size over 128 GB. Due to limitations of older JVMs, this was previously capped at 128 GB, even when the maximum heap size was larger. |
DS-13559 |
Fixed an issue that could cause the server to overlook attribute options in an indexed search filter. |
DS-13644 |
Fixed an issue with the Pass Through Authentication Plugin where if a DN map was not used, the controls from the native bind operation were included in the pass through authentication request. |
DS-13678 |
Updated the create-systemd-script tool by adding resource limits for available open file descriptors (NOFILE), and shared memory reservations (MEMLOCK). The generated script lists the recommended file descriptors limit and the resource limit setting for enabling large page support. The settings in the create-systemd-script output supersedes prior documentation for setting the number of open file descriptors on non-systemd systems. |
DS-13727 |
Added an Identify References virtual attribute provider. Virtual attributes of this type will have values that are the DNs of entries that contain a specified attribute with a value that matches the DN of the entry containing the virtual attribute. For example, this could be used to create a virtual directReports attribute whose values are the DNs of the entries that list the target user as their manager. |
DS-13748 |
Updated the ldif-diff tool to ensure that change records for delete operations will be ordered to ensure that a delete for a parent entry will never come before the deletes for its children. |
DS-13771 |
Updated the initial output of export-ldif to report that it is calculating a disk-ordered cursor rather than "Exported 0 entries." |
DS-13783 |
Updated the server to better utilize worker threads and reduce the potential for a work queue backlog when processing multiple concurrent long-running operations. |
DS-13794 | Updated the server to generate an administrative alert if it detects that a database environment was not closed cleanly and may require a time-consuming recovery process. |
DS-13820 |
Fixed an issue involving transactions sent through a PingDirectoryProxy Server with Entry Balancing configured. If the transaction contained requests that targeted entries that were not in the global index, then duplicate requests were included in the resulting Multi-Update operation forwarded to the Datastore. |
DS-13856 |
Updated the server to use the latest 6.4.9 release of Berkeley DB Java Edition. |
DS-13862 |
Updated the server to discourage conflicts between indexes and virtual attributes. A search that targets an indexed attribute will only identify entries with real values as potential matches, and may omit entries that match the filter if they match because of a virtually-generated value. |
DS-13899 |
Fixed an issue with large password history duration values that may have prevented some passwords in the history from being considered a match when a new password with the same value was added. |
DS-13983 |
Fixed an issue where dynamic group membership searches lead to resource and memory leaks. |
DS-14014 |
Updated the server to use the latest 6.4.12 release of the Berkeley DB Java Edition. This version addresses a possible data corruption bug in versions 6.3.0 to 6.4.11. |
DS-14036 |
Fixed an issue where password history values could be duplicated in replicated environments. |
DS-14060 |
Fix an issue in the SCIM interface where an attribute required by the SCIM schema could be deleted by a PATCH operation. |
DS-14074 |
Added the ability to protect Velocity templates using the basic authentication scheme. |
DS-14099 |
Fixed an issue where replication would stall while performing searches on LDAP referrals. |
DS-14116 |
Fixed issues pertaining to search references. One caused replication to stall, and the other caused multiple copies of the same reference to be returned in a single search. |
DS-14133 |
Fixed a couple of corner cases in which the server could treat smart referrals improperly. This includes search operations based above smart referrals contained in entries whose DNs contain escaped commas, as well as some search and compare operations based at least one level below a smart referral. |
DS-14140 |
The ldifsearch command now supports the option "---isCompressed" for LDIF files that have been compressed with gzip. |
DS-14259 |
Updated the email OTP delivery mechanism to allow retrieving email addresses from fields contained inside JSON objects, optionally using a JSON object filter to select which of several addresses should be used (e.g., only attempt to use verified email addresses). Similar changes have been made to the Twilio OTP delivery mechanism for obtaining phone numbers for SMS messages. |
DS-14311 |
Fixed a race condition that could arise from simultaneous attempts to add entries with an attribute value that would cause an exploded index key to exceed its index entry limit. |
DS-14349 |
Fixed an issue with the collect-support-data tool when using the --pid argument. Only one jstack was being collected, instead of using the amount specified by the --maxJstacks argument. |
DS-14398,DS-14399 |
Added password storage schemes that leverage the Bcrypt and scrypt key derivation functions. These storage schemes require the free and open source Bouncy Castle library, which is not included with the server. This library must be obtained from https://bouncycastle.org/ and placed in the server lib directory before these storage schemes can be used. |