Synchronization through a PingDirectoryProxy server overview

To handle data synchronization through a PingDirectoryProxy server, PingDirectory servers have a cn=changelog state management system that supports a token-based API.

In a standard, non-proxied configuration, PingDataSync polls the source server for changes, determines if a change is necessary, and fetches the full entry from the source. Then, it finds the corresponding entry in the destination endpoint using correlation rules and applies the minimal set of changes. The server fetches and compares the full entries to make sure it does not synchronize any stale data from the change log.

In a proxied environment, PingDataSync passes the request through a proxy server to the backend set of directory servers. PingDataSync uses the highest priority proxy server designated in its endpoint server configuration and can use others in the event of a failover.

The following figure illustrates a deployment with two endpoints consisting of a proxy server deployment in front of the backend set of directory servers.

A diagram illustrating synchronization with a SCIM sync destination.

Change log operations

When PingDataSync runs a poll for any changes, it sends a get change log batch extended request to the cn=changelog backend. The batch request looks for entries in the change log and asks for the server ID, change number, and replica state for each change.

The PingDirectoryProxy server routes the request to a directory server instance, which then returns a changed entry plus a token identifying the server ID, change number and replica state for each change. The PingDirectoryProxy server then sends a get change log batch response back to PingDataSync with this information. For entry-balancing deployments, the PingDirectoryProxy server must re-package the directory server tokens into its own proxy token to identify the specific data set.

The first time that PingDataSync issues the batch request, it also issues a get server ID request to identify the specific server ID that is processing the extended request. The PingDirectoryProxy server routes the request to the directory server instance, and then returns a server ID in the response. With the next request, PingDataSync sends a route to server request that specifies the server instance to access again in this batch session. It also issues a server ID request in the event that the particular server is down. This method avoids round-robin server selection and provides more efficient overall change processing.

PingDirectory server and PingDirectoryProxy server tokens

The PingDirectory server maintains a change log database index to determine when to resume sending changes (for ADD, MODIFY, or DELETE operations) in its change log. While a simple stand-alone directory server can track its resume point by the last change number sent, replicated servers or servers deployed in entry balancing environments have a different change number ordering in its change log because updates can come from a variety of sources.

The following figure illustrates two change logs in two replicated directory servers, server A and B. "A" represents the replica identifier for a replicated subtree in Server A, and "B" represents the replica identifier for the same replicated subtree in server B. The replica identifiers with a hyphen ("-") mark any local, non-replicated but different changes. While the two replicas record all of the changes, the two change logs have two different change number orderings because updates come in at different times.

A screen capture showing different change number order in two replicated change logs

To track the change log resume position, the PingDirectory server uses a change log database index to identify the latest change number position corresponding to the highest replication CSN number for a given replica. This information is encapsulated in a directory server token and returned in the get change log batch response to the PingDirectoryProxy server. The token has the following format:

Directory Server Token: server ID, changeNumber,replicaState

For example, if the PingDirectoryProxy server sends a request for any changed entries, and the directory servers return the change number 1003 from server A and change number 2005 from server B, then each directory server token would contain the following information:

Directory Server Token A:
  serverID A, changeNumber 1003, replicaState {15(A)}

Directory Server Token B:
  serverID B, changeNumber 2005, replicaState {12(B), 15(A)}

Change log tracking in entry balancing deployments

Change log tracking can become more complex because a shared area of data can exist above the entry-balancing base distinguished name (DN) in addition to each backend set having its own set of changes and tokens.

In the following figure, Server A belongs to an entry-balancing set 1, and server B belonging to an entry-balancing set 2. Shared areas that exist above the entry-balancing base DN are assumed to be replicated to all servers. "SA" represents the replica identifier for that shared area on Server A and "SB" represents the replica identifier for the same area on Server B.

The PingDirectoryProxy server cannot pass a directory server token from the client to the directory server and back again. In an entry-balancing deployment, the PingDirectoryProxy server must maintain its own token mechanism that associates a directory server token (changeNumber, replicaIdentifier, replicaState) to a particular backend set.

Proxy Token:
backendSetID 1: ds-token 1 (changeNumber, replicaIdentifier, replicaState)
backendSetID 2: ds-token 2 (changeNumber, replicaIdentifier, replicaState)

For example, if the PingDirectoryProxy server returned change 1002 from Server A and change 2002 from Server B, then the Proxy token would contain the following:

Proxy Token:
backendSetID 1: ds-token-1 {serverID A, changeNumber 1002, replicaState (5
(SA), 15(A)}
backendSetID 2: ds-token-2 {serverID B, changeNumber 2002, replicaState (10
(SB), 20(B)}

For each change entry returned by a backend, the PingDirectoryProxy server must also decide whether it is a duplicate of a change made to the backend set above the entry-balancing base. If the change is a duplicate, then it is discarded. Otherwise, any new change is returned with a new value of the proxy token.