Upgrade Considerations

Important considerations for upgrading to this version of the PingDirectoryProxy Server

  • Note: The product names have been updated to reflect the UnboundID acquisition by Ping Identity. This is a naming and branding change only; the code base is the same as in prior releases and will continue to be maintained into the future.

  • If upgrading the server that was running an older version of the JDK, run the dsjavaproperties --initialize command after the software upgrade to compare the settings of the older JDK with requirements for the new server software. Apply any necessary changes to the upgraded server based on previous performance settings.

  • The 6.0 release makes these changes to supported platforms:

    • CentOS 7.2 and RedHat 7.2 are now supported operating system versions.
    • SUSE 11 SP4 is now supported; SUSE 11 SP2 support has been retired.
    • Linux KVM and VMWare ESXi 6.x have been added as supported virtual machine environments.
    • Deprecation of JDKs 7.x. Customers are strongly advised to use JDK 8 with this release. A later release of the platform will remove support for JDK versions 7.x. At that time customers will be required to upgrade to JDK 8.x when upgrading servers. This will apply to all of our JDK flavors (OpenJDK, OracleJDK, IBM JDK) on all platforms.
    • WildFly 9.x (renamed from JBoss) is now a supported application server for all web applications, including the Administrative Console and sample applications. Support for JBoss 7.x has been retired. Tomcat support remains unchanged in this release.
  • PBKDF2 is now the default encoding for root passwords. This only affects new installations.

  • In addition to changing the default password storage scheme for root users to PBKDF2, the default password storage scheme for regular users has been changed to salted 256-bit SHA-2.

  • HTTPS defaults to ON: servers now default to use HTTPS for console and API connections including the SCIM API. This may affect automation scripts and development environments where HTTPS has not been in use before.

  • Generated user passwords, for example those created by the server during a password reset sequence, are now created as pass-phrases instead of random character strings. This makes them them easier to type and remember. This change will not affect upgrades.

  • The /config directory file permissions have been changed so that they are only accessible by the server user.

  • Customers who choose to use the optional encryption algorithms provided by the third-party BouncyCastle library are encouraged to upgrade to BouncyCastle 1.54.

What's New

These are new features for this release of the PingDirectoryProxy Server

  • Added a new control for very large result sets 'maximum-sort-size-limit-without-vlv-index,' which allows client applications to request that the server gracefully degrades to unsorted results in cases where sorting a very large result set would have caused a time-out.

  • Added LDAP support for applications that authenticate users with Yubikey one-time passwords. The extensions include the UNBOUNDID-YUBIKEY-OTP SASL handler configuration object, extended operations and command line tools for registering a user’s Yubikey device, deregistering, and supporting authentication using either the one-time password (OTP) only, or the OTP together with a static password. The server can be configured to use the public Yubico validation service, or a different validation service. The Yubikey FIDO U2F, OATH HOTP, and PGP modes are not supported.

  • Added new "generate TOTP shared secret" and "revoke TOTP shared secret" extended operations to make it easier for applications to enable TOTP authentication for users. While these operations are primarily intended to be invoked programmatically, a generate-totp-shared-secret tool can be used to invoke these operations from the command line.

  • A new transform-ldif tool is available to read an LDIF file and write an updated file with a number of changes applied. The transformations include:

    • Scramble, replace, redact, or exclude a specified set of attributes.
    • Replace values of a specified attribute with a generated value that includes a sequential counter.
    • Exclude entries matching a provided set of criteria.
    • Add a given set of attribute values to entries matching a provided set of criteria.
    • Rename attributes.
    • Replace the base DN for entries in a specified subtree.
  • A new load-ldap-schema-file tool is available for loading LDAP schemas while a server is active and on-line.

  • A new register-yubikey-otp-device tool is available for creating or changing associations between users and specific OTP devices.

  • The *rate performance testing tools now includes some additional sample rate pattern files: hockey stick, step-function, sine, triangle, sawtooth and square wave patterns.

  • The setup command now logs its input arguments, making it easier to confirm or duplicate a setup process. This changes the content of the log and may affect automated scripts that read these log files.

  • The config-diff tool, which makes it easy to compare and reconcile settings between server instances, now also supports the --pretty-print option which adds line breaks to the generated lists of dsconfig commands.

  • The manage-account tool has been enhanced significantly to make it easier to perform operations that affect large sets of user accounts including bulk lock-outs, parallel processing of updates, support for input filter criteria and DN lists. In particular, the manage-account tool now supports explicitly setting user accounts to the "locked-out" state. This is an improvement over earlier versions which required manipulation of operational attributes. See the command help for a complete list of the options and new sub-commands.

  • For easier consumption by third-party analysis tools, the Directory Servers and Directory Proxy Servers can now output JSON log formats. Similar support will be added to the Data Sync and Governance Brokers in a later release.

  • To help avoid issues when indexes near their index-entry-limit, the verify-indexes command now has the following two options:--listKeysNearestIndexEntryLimit, and --listKeysExceedingIndexEntryLimit. The Admin Guide includes a new section, "Monitoring Index Entry Limits", which explains how to set, track, and tune the server's Index Entry Limit values.

  • Monitor entries have been added for a number of related metrics, all of which can be set to trigger alarms:

    • ds-index-unique-keys-near-entry-limit-accessed-by-search-since-db-open
    • ds-index-unique-keys-exceeding-entry-limit-accessed-by-search-since-db-open
    • ds-index-unique-keys-near-entry-limit-accessed-by-write-since-db-open
    • ds-index-unique-keys-exceeding-entry-limit-accessed-by-write-since-db-open
  • The Pass-Through Authentication plugin has a new "allowLaxPassThroughAuthenticationPasswords" option that permits password changes that do not comply with the Directory Server's password policy. This facilitates integration in cases where the pass-through system has less-strict rules for new passwords.

  • For Java developers whose tools and workflows make use of Maven, the Server SDK jar has been deployed to Maven Central so that a developer can now add the Server SDK as a project dependency by adding a few lines to a project's pom.xml. Also, developers can now generate a Server SDK project that Maven-aware IDEs such as IntelliJ IDEA can package into an extension bundle with no special configuration needed. This benefit extends similarly to continuous integration systems such as Jenkins.

  • The dsconfig tool provides the ability to search for and quickly navigate to configuration objects and properties in which the name, synopsis, or description matches a provided pattern.

  • A new rotate-log tool and task have been added, which can be used to trigger rotation of one or more log files.

  • The Configuration API is now fully supported for all servers. In this release, the API was changed to match SCIM conventions for attribute naming, resource modeling, and the standard HTTP verbs. The UnboundID SCIM 2 SDK (available through GitHub) can now be used with the Configuration API.

  • All servers have an updated web Administrative Console, which includes:

    • New layouts for operational statistics, processing time, queues, all monitors, and the list of installed extensions.
    • Alert and alarm displays, summarizing the data in cn=alerts and cn=alarms and based on the configured gauges. Plus, filtering and searching for these.
    • A new LDAP Schema Editor for importing schema files, validity checking, creation and editing of object classes and attribute types. The editor also supports viewing of the attribute syntaxes, inheritance, and indexes that exist for each attribute and the dependencies between object classes and attributes.
  • The new Administrative Console can also be deployed to independent application servers instead of being co-hosted by the servers. This simplifies deployment models and increases separation between data and application layers.

  • To assist with situations where a very large number of changes may cause disk, memory, and server start time to increase unexpectedly, alerting and gauge features have been added to the Recent Changes Database.

  • Servers can now trigger events whenever log file rotation occurs. This includes "copy on rotate" and "summarize on rotate" listeners, as well as Server SDK support for creating custom log file rotation listeners.

  • It is now possible to create, change, and remove root user accounts across the topology using the dsconfig tool and Administrative Console.

Known Issues/Workarounds

The following are known issues in the current version of the PingDirectoryProxy Server

  • When deploying the Administrative Console in Tomcat 8, and accessing the Administrative Console application using Tomcat's Web Application Manager, some browsers (including Safari and Firefox) will generate a path URL that encodes the dash in ubid-console. This results in a path such as http://localhost:8888/ubid%2Dconsole/, which causes session management errors. To workaround this issue, copy and paste the generated link into a browser, and replace the encoded dash with a dash (-) character.

  • The PermSize and MaxPermSize JVM properties are no longer supported in JDK 8, and will safely be ignored. These properties can be removed by modifying the config/java.properties file and running "bin/dsjavaproperties" while the server is offline.

  • Security criteria for root passwords with the default configuration will be increased in a future release. This might affect automated installation scripts that currently use less secure passwords. This will not affect existing root accounts.

  • The dsconfig tool and the Administrative Console enables creating and managing new Root DN users in this release. However, there is a limitation with changing the password of the currently logged in administrator. The ldappasswordmodify command can be used to change the administrator's password by providing the current and new password.

Resolved Issues

The following issues have been resolved with this release of the PingDirectoryProxy Server:

Ticket ID Description
DS-979

Added the ability to search for configuration objects and their properties by name with the dsconfig tool.

DS-4235

Added support for log file rotation listeners, which allow for custom processing whenever a log file is rotated out of service so that the server will no longer write to it. A copy listener (which will copy the rotated log file to an alternate location, optionally compressing it in the process), and a summarize listener (which will invoke the summarize-access-log tool on the rotated log file) are included. The Server SDK also includes an API for creating custom log file rotation listeners.

DS-7017

Added support for authenticating with one-time passwords generated by YubiKey devices. The server may be configured to require static passwords in conjunction with YubiKey one-time passwords as a form of two-factor authentication, or it may be configured so that a one-time password alone is sufficient for authentication.

DS-7505,DS-13571,DS-13860

Updated the default file permissions for new installations on UNIX-based systems. Files and directories included in the zip file will be only be accessible to their owner (the user that unzipped the file) by default.

Newly-created files and directories will also be assigned permissions that allow them to be accessed only by the account used to run the server. Existing configuration options for setting file permissions (the log-file-permissions and db-directory-permissions properties) will continue to behave as before. The new config/server.umask file will control the default permissions for all other newly-created files and directories.

DS-9407,DS-15183,DS-15220

Updated the initial server configuration to improve security and usability. These changes apply only to new installations and will not be applied when updating an existing installation. Changes include:

  • Updated the default password policy to use a default password storage scheme that uses a salted 256-bit SHA-2 digest rather than a salted SHA-1 digest.
  • Updated the root password policy to use a default password storage scheme of PBKDF2 rather than salted 512-bit SHA-2.
  • Updated the secure password policy to use a default password storage scheme of PBKDF2 rather than a CRYPT variant that uses multiple rounds of 256-bit SHA-2.
  • Updated the password policy import plugin so that it will attempt to use the default password policy to select the password storage scheme(s) to use for entries that do not explicitly specify a password policy. The plugin will also fall back to using a salted 256-bit SHA-2 scheme instead of a salted SHA-1 scheme.
  • A number of weaker password storage schemes have been disabled by default, including base64, clear, unsalted MD5, salted MD5, 3DES, RC4, and unsalted SHA-1.
  • The default password policy has been updated to use a password generator that generates very strong yet memorable passphrases rather than a shorter and less-memorable string of randomly-selected characters.
  • Many of the server loggers have been updated to include additional log elements by default, including the instance name, requester DN, requester IP address, and request controls.
  • The exact match identity mapper has been updated to look at the mail attribute in addition to the uid attribute. When targeting a user with an authentication ID value (as when using SASL authentication or the proxied authorization v2 request control), it is now possible to specify an email address as an alternative to a user ID.
  • The UNBOUNDID-TOTP SASL mechanism handler has been updated to prevent TOTP password reuse by default.
  • Added new request criteria that make it possible to identify requests that target the root DSE or the subschema subentry. The global configuration has been updated so that requests targeting these entries will be in the default exceptions lists if the server is configured to reject insecure or unauthenticated requests.
  • Updated the template that setup generates for creating sample data to use a more logical and user-friendly numeric range. When the user requests N entries, setup would previously number the entries 0 through N-1 (for example, if the user requested 1000 entries, they would be numbered 0 through 999). It is logical for a user to expect them to be numbered 1 through 1000, but this change could break things that expecting to find an entry numbered with zero. To address this, if the user requests the server be populated with sample data, setup will create one more entry than actually requested so the numbering will go from 0 to N.
DS-10312

Added the server's process ID to the output of the status tool.

DS-10464

Added a new rotate-log tool to request the rotation of one or more log files.

DS-10466,DS-10765,DS-14479,DS-15318,DS-16154

Addressed a few issues in config-diff. In some situations, config-diff would not generate commands in an order that respected all dependencies. This has been fixed. Most expected warnings are now excluded by default but can be included in the output with the --includeAllWarnings option. The --sourceBindPasswordFile and --targetBindPassword are now applied in conjunction with the --targetConfigGroup and --sourceConfigGroup options.

DS-10946

Added a --dry-run option to dsconfig, which can be used in batch mode to validate the configuration changes in a batch file without applying them.

DS-11076

Updated the Directory Proxy Server to forward Password Policy State and Validate TOTP Password extended operations to the backend Directory Server for processing. Also, the Directory Proxy Server is now able to forward the following extended operations to a backend Directory Server in an entry-balancing configuration: Deliver Password Reset Token, Deliver Single Use Tokens and Get Supported OTP Delivery Mechanisms.

DS-12191

Added support for setting the request header size in the Jetty http configuration server properties.

DS-12555

Changed the Directory Proxy Server search behavior to fail when processing a critical RouteToBackendSetRequestControl with an unknown entry balancing request processor ID included. Non-critical controls will ignore unknown request processor IDs.

DS-13401

Improved the collect-support-data tool to include information provided by systemd on platforms that support it.

DS-13700

Updated command-line tools based on the LDAP SDK tool APIs to add the following features:

  • Tools can obtain default values for any arguments not provided on the command line from a properties file. If it exists, the server's config/tools.properties file will be used by default. Command-line arguments can be used to specify an alternate properties file or to indicate that no properties file should be used.
  • Tools can be launched in an interactive mode, in which the user is prompted for arguments used to establish and authenticate the connection, and for any other required arguments. The user can then use an interactive menu to specify values for any remaining arguments.
DS-13823

Collect-support-data tool now captures Kerberos config and log information.

DS-14213

Added the ability to create local constants in LDIF template files using the new 'local' keyword.

DS-14430

Updated the Apache commons collections library to address the security vulnerability described by CVE 2015-4852.

DS-14548

Added a monitor entry for each Server SDK extension.

DS-14694

Added a --prettyPrint option to the config-diff tool to make the output more human-readable.

DS-14699

The server now enforces that attributes referenced in configuration properties are defined explicitly in the local schema. This includes cached attribute types as well, such as, the attribute types that a Proxy Server caches from backend Directory Server instances.

DS-14704

Updated the restore command so that it can no longer be used to restore a backup of the config backend. The command now points the administrator for safer ways to revert configuration changes, including using config-diff.

DS-14749

Updated the server's support for the Twilio Messaging Service so that it uses the newer "Messages" API when sending SMS messages instead of the older "SMS" API. The older API has been deprecated, and Twilio now imposes a 120-character limit for messages sent via that API. The messages API allows the server to take advantage of the full 160 characters per SMS message.

DS-14805

Improved the admin alert health check system to handle situations where a massive number of admin alerts generated on an external server could result in the server being seen as unavailable.

DS-14807

Tools used to prepare a server for access by another server, such as prepare-external-server, now validate base DN entries before any modifications are performed on the prepared server.

DS-14857

Fixed an issue with the dsjavaproperties tool where java properties for PermSize and MaxPermSize could be added when using JDK 8, which no longer supports these options.

DS-14866

Fixed a race condition in the attribute index's cleanup thread, which was causing the Proxy Server to hang during startup.

DS-14878

Improved memory utilization when processing entries with very large attributes, to prevent possible data retention in memory.

DS-14889

Fixed an issue where the Proxy Server forwarded an atomic multi-update extended operation twice to the same backend set in an entry-balanced configuration, when the operation included a RouteToBackendSet control with absolute routing.

DS-14919

Added support for JSON-formatted access and error log messages.

DS-14922

Improved the subordinate subtree view processing logic to only use the exclude branch request control for subordinate views that are within the scope of the target DN, for search requests processed by the parent subtree view.

DS-14923

Updated the bcrypt, crypt, PBKDF2 and scrypt password storage schemes so they can be used to create new instances.

DS-14979

Fixed a case where attribute syntax configuration changes would not apply to undefined attributes, which rely on default attribute types.

DS-15015

Server SDK extensions are now built with a Java source version of 1.7 by default.

DS-15059

Fixed an issue with simple paged results that require spanning multiple backend servers through an entry balanced request processor to fill a single page. The operation would return with a 'no such object' result code if one of the backend servers used did not have any matching results.

DS-15088

The former suite of Administrative Console applications, each of which were tied to a particular product (for example the dsconsole.war for the Directory Server) are no longer available, and have been superceded by a new version of the Administrative Console capable of managing any server product. You can choose to access the Administrative Console by hosting it within a server, or by deploying it in an external servlet container. For the former, enable an HTTP Connection Handler and add the Administrative Console Web Application Extension to the handler. For the latter, download and unzip the management-console-[version].zip file, and install the ubid-console.war file according to your container's instructions.

DS-15108

Replaced the scramble-ldif tool with a more powerful transform-ldif tool with support for a number of additional transformation types. The new transform-ldif tool is backward compatible with the former scramble-ldif tool, and the scramble-ldif shell script and batch file are still included with the server to ensure compatibility with scripts that depend on that tool.

DS-15132

Improved the locking strategy for multi-update requests to better accommodate delete and add requests for the same entry. This also enables graceful failures for bad requests, instead of lock timeouts.

DS-15168

Increased the maximum size of the thread pool that is used to process entry balancing broadcast operations. The maximum is now 64 times the number of worker threads with an upper limit of 2048.

DS-15175

The Configuration API now returns unquoted, native Javascript values for integer, real number, and boolean properties. Duration and size property values, for example '1 w' or '100 G', continue to be represented as Javascript string types.

DS-15178

Improved the error messages and examples for create-rc-script and create-systemd-script by explicitly suggesting the use of sudo so that the scripts can modify protected files.

DS-15186

Fixed a rare condition where priming the Directory Proxy Server would not complete if a backend Directory Server becomes unavailable.

DS-15195

Added support for server affinity using extended operations.

DS-15221

Changed interactive setup default value for HTTPS enablement.

DS-15337

Improved the error messages for create-rc-script and create-systemd-script when the directory in which the script will be created does not exist.

DS-15349

Added support for a "generate TOTP shared secret" extended operation that allows a client to request that the server generate a shared secret for a specified user that will be stored in the user's entry and returned to the client. That shared secret can be used to generate time-based one-time passwords for use in the course of authenticating to the server through the UNBOUNDID-TOTP SASL mechanism. A "revoke TOTP shared secret" extended operation was also added to allow a shared secret to be eliminated if it is no longer needed or may have been compromised. The password policy state extended operation and the manage-account command-line tool have also been updated to provide support for manipulating the set of TOTP shared secrets for a user.

DS-15361,DS-15363,DS-15434

Updated interactive setup to display default values, and improved the overall layout and appearance.

DS-15400

Addressed an issue where dsconfig incorrectly allowed certain configuration objects to be deleted.

DS-15412

Improved the error messages produced by the manage-extensions tool when attempting to install invalid extensions.

DS-15417

Updated the global ACIs that ship with the server to use a separate ACI for each control or extended request to allow by default, rather than grouping all desired controls together in one ACI and all desired extended requests together in a second ACI. This change will only be reflected in new installations, and not when updating an existing deployment.

DS-15422

Root DN User configuration entries can now be fully managed through the configuration management interfaces such as dsconfig and the Administrative Console.

DS-15466

Added more logging information when initializing web application and servlet extensions in case an extension causes conflicts or delays.

DS-15521

Updated setup to encode the root password with the PBKDF2 password storage scheme instead of SSHA512.

DS-15522

Updater tool will increase PermSize and MaxPermSize parameters to recommended value to prevent Java JVM pauses.

DS-15559

Added support for an UNBOUNDID-EXTERNALLY-PROCESSED-AUTHENTICATION SASL mechanism that indicates that an application attempted to verify the identity of a user whose account is stored in the server but that used a form of authentication that is external to the server (for example, via social login). The server will not alter the authentication state of the underlying connection, but may veto a successful external authentication if the user's account is not in a usable state (for example, the account is locked or disabled, or the password is expired), or it may update password policy state for the user to reflect the authentication attempt (for example, updating the last login time and IP address for a successful authentication, or recording the failed attempt and potentially locking the account for an unsuccessful authentication).

DS-15571

Increase the minimum memory requirements for the server process from 256MB to 384MB to accommodate the Administrative Console.

DS-15576

Added a load-ldap-schema-file tool that will allow the server to recognize a new schema file, or an updated version of an existing schema file, and make the definitions immediately available without needing to restart the server.

DS-15592

Fixed an error that could occur during upgrade when the configuration can not be loaded due to missing custom schema.

DS-15621

Updated the Groovy Scripting Language version to 2.4.6.

DS-15622

Fixed an issue that prevented the deletion of disabled debug loggers.

DS-15789

Updated the server to allow users with expired passwords to authenticate with SASL mechanisms that do not involve passwords.

DS-15827

Updated the globally-unique attribute plugin so that the filter property applies to conflict searches, and matches entries being added or modified.

DS-15849,DS-15850,DS-15851,DS-15852,DS-15853

Rewrote the manage-account tool to provide many new features:

  • Added the following new subcommands:
    • get-account-is-usable
    • get-account-usability-notice-messages
    • get-account-usability-warning-messages
    • get-account-usability-errors-messages
    • get-account-is-not-yet-active
    • get-account-is-expired
    • get-password-is-expired
    • get-password-expiration-time
    • get-account-is-failure-locked
    • set-account-is-failure-locked
    • get-failure-lockout-time
    • get-account-is-idle-locked
    • get-idle-lockout-time
    • get-account-is-password-reset-locked
    • get-password-reset-lockout-time
    • get-account-activation-time
    • set-account-activation-time
    • clear-account-activation-time
    • get-seconds-until-account-activation
    • get-last-login-ip-address
    • set-last-login-ip-address
    • clear-last-login-ip-address
    • get-password-history-count
  • Added new ways to target multiple users with a single command. It was already possible to provide a file with the DNs of the users to target. There are now additional options for providing one or more search filters or user IDs to identify which users to target.
  • Added automatic retry support. If an operation fails in a manner that indicates the connection is no longer valid, the tool will retry the operation on a newly-created connection. It is also possible to provide multiple host name and port values to allow operations to be sent to multiple servers.
  • Added the ability to use multiple threads to operate more quickly when targeting multiple users.
  • Added the ability to limit the rate at which the tool operates. The target rate may be specified as a fixed number of operations per second, or it may vary over time.
  • Changed the output format so that the result of each operation is provided in an LDIF representation. The output remains easy for a person to read, but it is now much easier to consume programmatically.
  • Added the ability to send the output to a specified file instead of or in addition to standard output. Also added the ability to write a reject file that contains only information about those operations that were not completed successfully.
DS-15920

Improved the warnings given when the maximum memory that all server components can consume is greater than the available memory in the JVM.

DS-15943

Addressed an issue where jsonObjectFilterExtensibleMatch queries in the proxy would fail if any DN Mappers were configured.

DS-16053

Improved the way we handle unexpected errors and invalid DNs during proxy transformations.

DS-16104

Updated the password policy state extended operation and the manage-account tool to provide a way to obtain a list of the SASL mechanisms and OTP delivery mechanisms that are available to a user, to determine whether a user has a TOTP shared secret, and to retrieve and manipulate the set of public IDs for the YubiKey OTP devices registered for a user.

DS-16224

Updated the sanitize-log tool to add support for JSON-formatted access and error log files.