Troubleshooting synchronization failures

While many PingDataSync issues are deployment-related and are directly affected by the hardware, software, and network structure used in the synchronization topology, most failures usually fall into one of the following categories:

Entry Already Exists: When an add operation was attempted on the destination server, an entry with the same distinguished name (DN) already exists.
No Match Found: A match was not found at the destination based on the current Sync Classes and correlation rules (DN and attribute mapping). When this value has a high count, correlation rule problems are likely.
Failure at Resource: Indicates that some other error happened during the synchronization process that does not fall into the first two categories. Typically, these errors are communication problems with a source or destination server.

Statistics for these and other types of errors are kept in the cn=monitor branch and can be viewed directly using the status command.

Troubleshooting "Entry Already Exists" failures

About this task

If there is a count for the EntryAlready Exists statistic using the status tool, verify the problem in the sync log. For example, the status tool displays the following information:

           --- Ops Completed for 'DS1 to DS2' Sync Pipe ---
Op Result              : Count
-----------------------:------
Success                : 0
Out Of Scope           : 0
Op Type Not Synced     : 0
No Change Needed       : 0
Entry Already Exists   : 1
No Match Found         : 1
Multiple Matches Found : 0
Failed During Mapping  : 0
Failed At Resource     : 0
Unexpected Exception   : 0
Total                  : 2

Verify the change by viewing the <server-root>/logs/sync file to see the specific operation, which could be due to someone manually adding the entry on the target server:

op=2 changeNumber=529277 replicationCSN=00000128AD0D9BA01E960008137D
replicaID=7830
pipe="DS1 to DS2" class="DEFAULT" msg="Detected ADD of
uid=user.1001,ou=People,
dc=example,dc=com at ldap://server1.example.com:1389, but cannot create this
entry at the destination because an equivalent entry already exists at
ldap://server3.
example.com:3389. Details: Search using [search-criteria dn:
uid=user.1001,ou=People,
dc=example,dc=com attrsToGet: [*, dn]] returned results;
[uid=user.1001,ou=People,
dc=example,dc=com]. "

Perform the following steps to troubleshoot this type of problem:

Steps

Assuming that a possible DN mapping is ill-formed, first run the ldap-diff utility to compare the entries on the source and destination servers. Then look at the ldap-diff results with the mapping rules to determine shy the original search did not find a match.

$ bin/ldap-diff \
  --outputLDIF config-difference.ldif \
  --baseDN "dc=example,dc=com" \
  --sourceHost server1.example.com \
  --targetHost server2.example.com \
  --sourcePort 1389 \
  --targetPort 3389 \
  --sourceBindDN "cn=Directory Manager" \
  --sourceBindPassword password \
  --searchFilter "(uid=1234)"

Review the destination server access logs to verify the search and filters used to find the entry. Typically, the key correlation attributes are not synchronized.
If the mapping rule attributes are not synchronized, review the Sync Classes and mapping rules, and use the information from the ldap-diff results to determine why a specific attribute may not be getting updated. Some questions to answer are as follows:
- Is there more than one Sync Class that the operation could match?
- If using an include-base-dn or include-filter in the mapping rules, does this exclude this operation by mistake?
- If using an attribute map, are the mappings correct? Usually, the cause of this error is in the destination mapping attribute settings. For example, if a set of correlation attributes is defined as: dn, mobile, accountNumber, and the accountNumber changes for some reason, future operations on this entry will fail. To resolve this, either remove accountNumber from the rule, or add a second rule as: dn,mobile. The second rule is used only if the search using the first set of attributes fails. In this case, the entry is found and the accountNumber information is updated.
If deletes are being synchronized, check to see if there was a previous delete of this entry that was not synchronized properly. In some cases, simpler logic should be used for deletes due to the available attributes in the change logs. This scenario could cause an entry to not be deleted for some reason, which would cause an issue when a new entry with the same DN is added later. Use this information for mapping rules to see why the original search did not find a match.
Look at the destination directory server access logs to verify the search and filters it used to find the entry. Typically, the key attribute mappings are not synchronized.

Troubleshooting "No Match Found" failures

About this task

If there is a count for the No Match Found statistic using the status tool, verify the problem in the sync log. For example, if the status tool displays the following:

           --- Ops Completed for 'DS1 to DS2' Sync Pipe ---
Op Result              : Count
-----------------------:------
Success                : 0
Out Of Scope           : 0
Op Type Not Synced     : 0
No Change Needed       : 0
Entry Already Exists   : 1
No Match Found         : 1
Multiple Matches Found : 0
Failed During Mapping  : 0
Failed At Resource     : 0
Unexpected Exception   : 0
Total                  : 2

Verify the change in the <server-root>/logs/sync file to see the specific operation:

[12/May/2016:10:30:45 -0500] category=SYNC severity=MILD_WARNING
msgID=1893793952
op=4159648 changeNumber=6648922 replicationCSN=4beadaf4002f21150000
replicaID=8469-
ou=test,dc=example,dc=com pipe="DS1 to DS2" class="Others" msg="Detected
DELETE of
'uid=1234,ou=test,dc=example,dc=com' at ldap://server1.example.com:389, but
cannot
DELETE this entry at the destination because no matching entries were found at
ldap://
server2.example.com:389. Details: Search using [search-criteria dn:
uid=1234,ou=test,dc=alu,dc=com filter: (nsUniqueId=3a324c60-5ddb11df-80ffe681-
717b93af) attrsToGet: [*, accountNumber, dn, entryuuid, mobile, nsUniqueId,
object-
Class]] returned no results."

Perform the following steps to fix the issue:

Steps

Test the search using the filter in the error message, if displayed. For example, if the sync log specifies filter: (nsUniqueId=3a324c60-5ddb11df-80ffe681-717b93af), use the ldapsearch tool to test the filter. If it is successful, is there anything in the attribute mappings that would exclude this from working properly?
Test the search using the full DN as the base. For example, use ldapsearch with the full DN (uid=1234,ou=People,dc=example,dc=com). If it is successful, does the entry contain the attribute used in the mapping rule? If the attribute is not in the entry, determine if there is a reason why this value was not synchronized. Look at the attribute mappings and the filters used in the Sync Classes.

Troubleshooting "Failed at Resource" failures

About this task

If there is a count for the "Failed at Resource" statistic using the status tool, verify the problem in the sync log. For example, if the status tool displays the following information:

             --- Ops Completed for 'DS1 to DS2' Sync Pipe ---
Op Result              : Count
-----------------------:------
Success                : 0
Out Of Scope           : 0
Op Type Not Synced     : 0
No Change Needed       : 0
Entry Already Exists   : 0
No Match Found         : 0
Multiple Matches Found : 0
Failed During Mapping  : 0
Failed At Resource     : 1
Unexpected Exception   : 0
Total :1

This will register after a change is detected at the source in any of the following cases:

If the fetch of the full source entry fails. The entry exists but there is a connection failure, server down, timeout, or something similar.
If the fetch of the destination entry fails or if the modification to the destination fails for an exceptional reason (but not for "Entry Already Exists," "Multiple Matches Found," or "No Match Found" issues).

Verify the change by viewing the <server-root>/logs/sync file to see the specific operation. If any of the following result codes are listed , the server is experiencing timeout errors:

resultCode=timeout: errorMessage=A client-side timeout was encountered while waiting 60000ms for a search response from server server1.example.com:1389
resultCode=timeout: errorMessage=An I/O error occurred while trying to read the response from the server
resultCode=server down: errorMessage=An I/O error occurred while trying to read the response from the server
resultCode=server down: errorMessage=The connection to server server1.example.com:1389 was closed while waiting for a response to search request SearchRequest
resultCode=object class violation: errorMessage='Entry device=1234,dc=example,dc=com violates the Directory Server schema configuration because it contains undefined object class

With these "Failure at Destination" timeout errors, look at the following settings in the PingDirectory server to determine if adjustments are needed:

Steps

For External Server Properties, check the connect-timeout property. This property specifies the maximum length of time to wait for a connection to be established before giving up and considering the server unavailable.

For the Sync Destination/Sync Source Properties, check the response-timeout property. This property specifies the maximum length of time that an operation should be allowed to be blocked while waiting for a response from the server. A value of zero indicates that there should be no client-side timeout. In this case, the server’s default will be used.

$ bin/dsconfig --no-prompt --port 389 \
  --bindDN "cn=Directory Manager" \
  --bindPassword password list-external-servers \
  --property connect-timeout

External Server         : Type             : connect-timeout : response-
                                                               timeout
------------------------:------------------:-----------------:-----------
server1.example.com:389 : sundsee-ds       : 10 s            : -
server2.example.com:389 : sundsee-ds       : 10 s            : -
server3.example.com:389 : ping-identity-ds : 10 s            : -
server4.example.com:389 : ping-identity-ds : 10 s            : -

For Sync Pipe Properties, check the max-operation-attempts, retry-backoff-initialwait, retry-backoff-max-wait, retry-backoff-increase-by, retry-backoff-percentage-increase. These Sync Pipe properties provide tuning parameters that are used in conjunction with the timeout settings. When a Sync Pipe experiences an error, it will use these settings to determine how often and quickly it will retry the operation.
```
$ bin/dsconfig --no-prompt list-sync-pipes \
  --property max-operation-attempts --property retry-backoff-initial-wait
\
  --property retry-backoff-max-wait --property retry-backoff-increase-by \
  --property retry-backoff-percentage-increase \
  --port 389 --bindDN "cn=Directory Manager" \
  --bindPassword password
```