PingDirectory

PingDataSync operations

PingDataSync provides seamless integration between disparate systems to transform data using attribute and DN mappings. A bulk resynchronization operation can be run to verify mappings and test synchronization settings.

Real-time synchronization

Real-time synchronization is performed with the realtime-sync utility. The realtime-sync utility polls the source server for changes and synchronizes the destination entries immediately. After the server determines that a change should be synchronized, it fetches the full entry from the source. It then searches for the corresponding entry in the destination endpoint using correlation rules and applies the minimum set of changes to synchronize the attributes. The server fetches and compares the full entries to make sure it does not synchronize any old data from the change log.

After a synchronization topology is configured, run resync to synchronize the endpoints, and then run realtime-sync to start global synchronization.

The realtime-sync tool is used for the following tasks:

  • Start or stop synchronization globally or for specific sync pipes only.

  • Set a start point at which synchronization should begin such as the beginning or end of the change log, at a specified change number, at a specified change sequence number, or at a specified time frame in the change log.

Data transformations

Data transformations alter the contents of synchronized entries between the source and destination directory server to handle variances in attribute names, attribute values, or DN structures. When entries are synchronized between a source and a destination server, the contents of these entries can be changed using attribute and DN mappings, so that neither server needs be aware of the transformations.

  • Attribute Mapping – Any attribute in the entry can be renamed to fit the schema definitions from the source endpoint to the destination endpoint. This attribute mapping makes it possible to synchronize information stored in one directory’s attribute to an attribute with a different name in another directory server, or to construct an attribute using portions of the source attribute values.

  • DN Mapping – Any DNs referenced in the entries can be transparently altered. This mapping makes it possible to synchronize data from a topology that uses one DIT structure to a system that uses a different DIT structure.

Bulk resync

The resync tool performs a bulk comparison of entries on source topologies and destination topologies. PingDataSync streams entries from the source, and either updates the corresponding destination entries or reports those that are different. The resync utility resides in the /bin folder (UNIX or LINUX) or \bat folder (Windows), and can be used for the following tasks:

  • Verify that the two endpoints are synchronized after an initial configuration.

  • Initially populate a newly configured target endpoint.

  • Validate that the server is behaving as expected. The resync tool has a --dry-run option that validates that synchronization is operating properly, without updating any entries. This option also can be used to check attribute or DN mappings.

  • Perform scheduled synchronization.

  • Recover from a failover by resynchronizing entries that were modified since the last backup was taken.

The resync tool also enables control over what can be synchronized, such as:

  • Include or exclude any source and destination attributes.

  • Apply an LDAP filter to only sync entries created since the last time the tool ran.

  • Synchronize only creations or only modifications.

  • Change the logging verbosity.

  • Set a limit on resync operations (such as 2000 operations per second) to reduce impact on endpoint servers.

The sync retry mechanism

PingDataSync is designed to quickly synchronize data and attempt a retry should an operation fail for any reason. The retry mechanism involves two possible retry levels, which are configurable on the Sync Pipe configuration using advanced Sync Pipe properties.

First Level Retry

If an operation fails to synchronize, the server will attempt a configurable number of retries. The total number of retry attempts is set in the max-operation-attempts property on the Sync Pipe. The property indicates how many times a worker thread should retry the operation before putting the operation into the second level of retry, or failing the operation altogether.

Second Level Retry

Once the max-operation-attempts property has been exceeded, the retry is sent to the second level, called the delayed-retry queue. The delayed-retry queue uses two advanced Sync Pipe properties to determine the number of times an operation should be retried in the background after a specified delay.

Operations that make it to this level will be retried after the failed-op-background-retrydelay property (default: 1 minute). Next, PingDataSync checks the max-failedop-background-retries property to determine the number of times a failed operation should be retried in the background. By default, this property is set to 0, which indicates that no background retry should be attempted, and that the operation should be logged as failed.

Background operations can hold up processing other changes, since PingDataSync will only process up to the next 5000 changes while waiting for a retried operation to complete.

Retry can be controlled by the custom endpoint based on the type of error exception. When throwing an exception, the endpoint code can signal that a change should be aborted, retried a limited number of times, or retried an unlimited number of times. Some errors, such as endpoint server down, should be retried indefinitely.

If the max-failed-op-background-retries property has been exceeded, the retry is logged as a failure and appears in the sync and the sync-failed-ops logs.