What's New

These are new features for this release of PingDataSync Server:

  • New features for data encryption in transit and at rest: added support for TLS 1.3, ability to encrypt and automatically decrypt sensitive files such as tools.properties and keystore pin files using the server data encryption keys, and the ability to more easily and securely separate master keys from data encryption keys by protecting the server encryption settings database using either Amazon Key Management Service (AWS KMS) or HashiCorp Vault.

  • Added support for Amazon Corretto JDK 8, Windows Server 2019, Red Hat Enterprise Linux 7.6, CentOS 7.6, Amazon Linux 2, and Docker 18.09.0 on Ubuntu 18.04 LTS.

  • More configuration flexibility. Brand new JSON Attribute Mapping provides a more natural way to construct JSON-based attributes rather than using Constructed Attribute Mapping. Base64 encoding and decoding options are added to allow for synchronizing binary data. LDAP extensible match and approximate match filters can be used to better select changes that require synchronization. Rate limiting has been added per sync pipe using the max-rate-per-second setting.

  • Publish changes to Kafka. Sync pipes in notification mode can now be configured to publish changes in PingDirectory to topics in Kafka. Messages are published to Kafka for additions, deletions, modifications, and for resynchronization. Each message contains the JSON-serialized format of the mapped attributes per LDAP object, before and after modification, if applicable.

Resolved Issues

The following issues have been resolved with this release of the Data Sync Server:

Ticket ID Description
PDSTAGING-570,DS-38334

The following enhancements were made to the topology manager to make it easier to diagnose the connection errors described in PDSTAGING-570:

- Added monitoring information for all the failed outbound connections (including the time since it's been failing and the last error message seen when the failure occurred) from a server to one of its configured peers and the number of failed outbound connections.

- Added alarms/alerts for when a server fails to connect to a peer server within a configured grace period.

PDSTAGING-570,DS-38344

The topology manager will now raise a mirrored-subtree-manager-connection-asymmetry alarm when a server is able to establish outbound connections to its peer servers, but those peer servers are unable to establish connections back to the server within the configured grace period. The alarm is cleared as soon as there is connection symmetry.

PDSTAGING-570,DS-38335

The dsreplication tool has been fixed to work when the node being used to enable replication is currently out-of-sync with the topology master.

DS-15734

Added a cipher stream provider that can be used to protect the contents of the encryption settings database with a key from the Amazon Key Management Service.

DS-18060

Added an HTTP servlet extension that can be used to retrieve the server's current availability state. It accepts any GET, POST, or HEAD request sent to a specified endpoint and returns a minimal response whose HTTP status code may be used to determine whether the server considers itself to be AVAILABLE, DEGRADED, or UNAVAILABLE. The status code for each of these states is configurable, and the response may optionally include a JSON object with an "availability-state" field with the name of the current state.

Two instances of this servlet extension are now available in the default configuration. A request sent to /available-state will return an HTTP status code of 200 (OK) if the server has a state of AVAILABLE, and 503 (Service Unavailable) if the server has a state of DEGRADED or UNAVAILABLE. A request sent to the /available-or-degraded-state will return an HTTP status code of 200 for a state of AVAILABLE or DEGRADED, and 503 for a state of UNAVAILABLE. The former may be useful for load balancers that you only want to have route requests to servers that are fully available. The latter may be useful for orchestration frameworks if you wish to destroy and replace any instance that is completely unavailable.

DS-36269

Added support for using Population names both as a default when configuring the PingOne Customer Sync Destination and in attribute mapping.

DS-36474

Updated the ldapdelete command-line tool to improve robustness and add features. Some of the new features include support for client-side subtree delete, deleting entries that match search filters, following referrals, writing failures to a rejects file, rate limiting, and support for a variety of additional controls.

DS-37617

HTTP Connection Handlers now accept client-provided correlation IDs by default. To adjust the set of HTTP request headers that may include a correlation ID value, change the HTTP Connection Handler's correlation-id-request-header property.

DS-37839

Make Fingerprint Certificate Mapper and Subject DN to User Attribute Certificate Mapper disabled by default on fresh installations. This will not affect upgrades from installations where these mappers are enabled.

DS-37959

Added support for insignificant configuration archive attributes.

The configuration archive is a collection of the configurations that have been used by the server at some time. It is updated whenever a change is made to data in the server configuration, and it is very useful for auditing and troubleshooting. However, because the entries that define root users and topology administrators reside in the configuration, changes to those entries will also cause a new addition to the configuration archive. This is true even for changes that affect metadata for those entries, like updates to the password policy state information for one of those users. For example, if last login time tracking is enabled for one of those users, especially with high-precision timestamps, a new configuration may be generated and added to the configuration archive every time that user authenticates to the server. While it is important for this information to be persisted, it is not as important for it to be part of the server's configuration history.

This update can help avoid the configuration archive from storing information about updates that only affect this kind of account metadata. If a configuration change only modifies an existing entry, and if the only changes to that entry affect insignificant configuration archive attributes, then that change may not be persisted in the server's configuration archive.

By default, the following attributes are now considered insignificant for the purpose of the configuration archive:

* ds-auth-delivered-otp * ds-auth-password-reset-token * ds-auth-single-use-token * ds-auth-totp-last-password-used * ds-last-access-time * ds-pwp-auth-failure * ds-pwp-last-login-ip-address * ds-pwp-last-login-time * ds-pwp-password-changed-by-required-time * ds-pwp-reset-time * ds-pwp-retired-password * ds-pwp-warned-time * modifiersName * modifyTimestamp * pwdAccountLockedTime * pwdChangedTime * pwdFailureTime * pwdGraceUseTime * pwdHistory * pwdReset

DS-38050

Updated the server to support encrypting the contents of the PIN files needed to unlock certificate key and trust stores. If data encryption is enabled during setup, then the default PIN files will automatically be encrypted.

Also, updated the command-line tool framework so that the tools.properties file (which can provide default values for arguments not provided on the command line), and passphrase files (for example, used to hold the bind password) can be encrypted.

DS-38072

Updated the server to enable TLSv1.3 by default on JVMs that support it (Java 11 and higher).

DS-38085

Fixed an issue in the installer where the Administrative Console's trust store type would be incorrectly set if it differed from the key store type.

DS-38090,DS-38564,DS-38567

The response header used for correlation IDs may now be set at the HTTP Servlet Extension level using the correlation-id-response-header configuration property. If set, this property overrides the HTTP Connection Handler's correlation-id-response-header property.

DS-38098

Added a max-rate-per-second configuration property to the Sync Pipe, which can be used to throttle the rate that changes are applied to the destination.

DS-38100

Added a Kafka Sync Destination, which can be used to publish changes from PingDirectory Sync Sources to Kafka topics. Changes are represented as the entry contents before and the entry contents after the change in JSON format.

DS-38109

Added the --skipHostnameCheck command line option to the setup script, which bypasses validation of the provided host name for the server.

DS-38253

Added the PingOne for Customers Sync Source to Ping Data Sync. Identities can now be synchronized from PingOne for Customers to on premises identity stores.

DS-38403

Fixed an issue that could prevent certain types of initialization failures from appearing in the server error log by default.

DS-38460

Added 'base64-encode-value' and 'base64-decode-value' properties to direct attribute mappings to facilitate synchronizing binary data.

DS-38495

Updated the PingDataSync Server to support LDAP filters in Sync Classes that include extensible match or approximate match components. The Data Sync Server supports the same matching rules as the PingDirectory Server.

DS-38512

Added a cipher stream provider that can be used to protect the contents of the encryption settings database with a secret passphrase obtained from a HashiCorp Vault instance.

DS-38550

Fixed an issue in which backups of the encryption settings database could be encrypted with a key from the encryption settings database.

DS-38582

Added attribute-comparison-method configuration option to SyncClass to allow syncing with a syntax-based comparison (default mode) or a byte-for-byte comparison method.

DS-38670

Fixed a bug where the startIndex value for SCIM requests would be incorrect if the used LDAPSearch element had more than one baseDN defined in the scim-resources XML file.

DS-38737

Fixed an issue where inter-server bind requests would fail if the cipher used reported a maximum unencrypted block size of 0.

DS-38833

Added JSON Attribute Mapping and JSON Attribute Mapping Field configuration objects that simplify constructing destination JSON attribute values from source attributes. Compared to using a Constructed Attribute Mapping to build up JSON, using these configuration objects does not requiring providing the JSON structure (braces, colons, and quotes), does not require escaping string values, and automatically omits fields whose values cannot be constructed.

DS-38849

Added an indent-ldap-filter tool that can make it easier to visualize the structure and components of a complex search filter.

DS-38864

Changed the default value of the HTTP Configuration property include-stack-traces-in-error-pages from 'true' to 'false'. Disabling this property prevents information about exceptions thrown by servlet or web application extensions from being revealed in HTTP error responses.

DS-38897,DS-38908

Fixed two issues in which the server could have exposed some clear-text passwords in files on the server file system.

* When creating an encrypted backup of the alarms, alerts, configuration, encryption settings, schema, tasks, or trust store backends, the password used to generate the encryption key (which may have been obtained from an encryption settings definition) could have been inadvertently written into the backup descriptor. This problem does not affect local DB backends (like userRoot), the LDAP changelog backend, or the replication database.

* When running certain command-line tools with an argument instructing the tool to read a password from a file, the password contained in that file could have been written into the server's tool invocation log instead of the path to that file. Affected tools include backup, create-initial-config, create-initial-proxy-config, dsreplication, enter-lockdown-mode, export-ldif, import-ldif, ldappasswordmodify, leave-lockdown-mode, manage-tasks, manage-topology, migrate-ldap-schema, parallel-update, prepare-endpoint-server, prepare-external-server, realtime-sync, rebuild-index, re-encode-entries, reload-http-connection-handler-certificates, reload-index, remove-defunct-server, restore, rotate-log, and stop-server. Other tools are not affected. Also note that this only includes passwords contained in files that were provided as command-line arguments; passwords included in the tools.properties file, or in a file referenced from tools.properties, would not have been exposed.

In each of these cases, the files would have been written with permissions that make their contents only accessible to the system account used to run the server. Further, while administrative passwords may have been exposed in the tool invocation log, neither the passwords for regular users, nor any other data from their entries, should have been affected. We have introduced new automated tests to help ensure that such incidents do not occur in the future.

We recommend changing any administrative passwords you fear may have been compromised as a result of this issue. If you are concerned that the passphrase for an encryption settings definition may have been exposed, then we recommend creating a new encryption settings definition that is preferred for all subsequent encryption operations, exporting your data to LDIF, and re-importing so that it will be encrypted with the new key. You also may wish to re-encrypt or destroy any existing backups, LDIF exports, or other data encrypted with a compromised key, and you may wish to sanitize or destroy any existing tool invocation log files that may contain clear-text passwords.

DS-38913

Added a set of message types to Trace Log Publishers that records events related to access token validation.

DS-38957

Fixed an issue that would throw an exception when trying to delete an entry containing uncached attributes if the LDAP changelog was enabled and using reversible form.

DS-39014

Addressed an issue in DataSync where LDAP operational attributes (such as isMemberOf) used in a SyncClass include-filter would not be requested from the source LDAP server if the LDAP filter was a nested filter.

DS-39086

Removed the version information page from the docs/build-info.txt endpoint. This information is now available in build-info.txt, which is located in the root directory.

DS-39174

Addressed an issue where a resync command could fail with a SEVERE_ERROR condition when run with multiple passes and the Sync Source was either Active Directory or PingOne for Customers.

DS-39177

Addressed an issue where a resync operation might not synchronize all users if a PingOne Customer Sync Source was configured with a value for population-to-synchronize.