Configuring assured replication

The PingDirectory server’s replication mechanism is based on the eventual-consistency model, which is a loosely-connected topology that propagates updates to other servers without waiting for their corresponding replication response messages.

There is a small window of time where updates are not all present on the replicas while the changes are replicating through the system. However, there are deployments that require immediate access to replicated data. In such cases, administrators can configure assured replication, which ensures that replication has completed before the update response is returned to the client.

From the LDAP client’s perspective, assured replication has no bearing on the result code of the operation, just on the time in which it takes to receive the response for those requests in which replication assurance is applied. Detailed information regarding assurance processing is available to an LDAP client with awareness of the assured replication control.

The assured replication mechanism uses server location to distinguish between local and remote servers to allow different policies to apply. For example, a common assurance approach is to respond to a client update after all servers in the same location have applied the update and one or more servers in remote locations have received the update. The level of assurance applied to each operation can be explicitly requested by the client, specified by the server configuration using the Replication Assurance Policy, or both.

Assured replication is supported by client requests directly to PingDirectory server, through a PingDirectoryProxy server, or both.

About the Replication Assurance Policy

Assured replication uses a Replication Assurance Policy to define the properties needed to ensure that replication has completed before the update response is returned to the client. Multiple policies can be defined, but only one policy is matched with a client update request. Each policy contains an evaluation order index that, together with an optional request and connection criteria, provides flexibility in matching a policy to request.

The Replication Assurance Policy defines local and remote assurance levels. A level defines how rigorous the policy should be when waiting for propagation of updates while local and remote distinguish servers in the same location versus servers in remote locations. Although optional, you should associate request or connection criteria with a policy to apply replication assurance appropriately.

PingDirectory server contains a default Replication Assurance Policy that is enabled with the local-level processed-all-servers assurance level. In addition to using the Default Replication Assurance Policy, any number of policies can be created and enabled. When a client request is received, the server iterates through the list of enabled policies according to each policy’s evaluation-order-index property. A smaller evaluation-order-index value (such as 1) has precedence over policies with larger evaluation-order-index values (such as 2, 3, 4, and so forth). The evaluation-order-index values do not need to be contiguous. The first policy that matches a request is associated with the operation.

The Replication Assurance Policy, which is defined on the PingDirectory server and not on the PingDirectoryProxy server, has the following properties:

evaluation-order-index

Determines the evaluation order among multiple Replication Assurance Policies that match against an operation. The smaller value has precedence.

local-level

Specifies the assurance level used to replicate to local servers. A local server is defined as a server with the same location property value as set in the global configuration. The local-level property must be set to an assurance level as strict as the remote-level property.

For example, if the remote-level is set to processed-all-remote-locations, then the local-level property must be processed-all-servers.

None: Replication to any local server is not assured.
received-any-server: At least one available local server must receive a replication update before a response is sent to the client.
processed-all-servers: All available local servers must complete replay of the replication update before the response is sent to the client. If a singular server is enabled, or only one server is available for a particular location, processed-all-servers returns a value of false.

remote-level

Specifies the assurance level used to replicate to remote servers. A remote server is defined as a server that has a different location property value as set in the global configuration.

None: Replication to any remote server is not assured.
received-any-remote-location: At least one server at any available remote location must receive a replication update before a response is sent to the client.
received-all-remote-locations: All available remote servers must receive a replication update before the response is sent to the client.
processed-all-remote-servers: All available servers at all locations must complete replay of the replication update before the response is sent to the client. If a single server is enabled, or only one server is available for a particular location, processed-all-remote-servers returns a value of false.

timeout

Specifies the maximum length of time to wait for the replication assurance requirements to be met before timeout and replying to the client.

connection-criteria

Specifies connection criteria used to indicate which operations from clients matching this criteria use this policy. If both connection criteria and request criteria are specified for a policy, then both must match an operation for the policy to be assigned.

request-criteria

Specifies request criteria used to indicate which operations from clients matching this criteria use this policy. If both connection criteria and a request criteria are specified for a policy, then both must match an operation for the policy to be assigned.

Servers in a replication topology are not required to share a homogeneous set of policies. You can configure the Replication Assurance Policies differently on the replicas in a topology. For example, if you configure server A to match add operations to a processed-all-servers assurance level, and server B to match add operations to a local received-any-server level, then add operations received on server A have the assurance level of processed-all-servers and add operations received on server B have the assurance level of received-any-server.

About assured replication

When implementing assured replication, make note of the following.

Item Description

Item	Description
Client Controls	The client can include an assured replication request control with each operation. This control allows the client to specify bounds on assurance levels, override the timeout assigned by the matching replication assurance policy for the associated operation, or both. The server always honors these request controls. For more information, see About the Assured Replication Controls.
Directory Proxy Server	Replication assurance policies are not supported on PingDirectoryProxy. Replication client controls are passed through to the underlying PingDirectory server backend.
Schema Backend Replication	The schema backend is not supported by assured replication. Replication assurance policies that include criteria to match this backend are rejected.
Backward Compatibility	Server versions that support assured replication are backwards-compatible with prior versions that do not support assured replication.
WAN-Friendly Replication	Assured replication functions independently from WAN-Friendly Replication and the notion of WAN Gateways.
Global Configuration Properties	PingDirectory server provides two configurable global configuration properties that determine the timing of the assurance source and maximum number of replication backup updates recognized as an available source. `replication-assurance-source-timeout-suspend-duration` Specifies the amount of time a replication assurance source is suspended from assurance requirements if it experiences an assurance timeout. While suspended, the source is excluded from assurance requirements for all operations originating on this PingDirectory server. This avoids the situation of repeated timeouts caused by degraded or offline servers. The default time is 10 seconds. `replication-assurance-source-backlog-fast-start-threshold` Specifies the maximum number of replication backlog updates a replication assurance source can have and be immediately recognized as an available source. If a source connects to this PingDirectory server with more than the configured threshold backlog updates, it is excluded from assurance requirements for all operations originating from the server until it completes at least one assurance successfully, indicated by the server receiving an update acknowledgment message from it within the timeout window. The default maximum number of updates is 1000.

Client Controls

The client can include an assured replication request control with each operation. This control allows the client to specify bounds on assurance levels, override the timeout assigned by the matching replication assurance policy for the associated operation, or both. The server always honors these request controls.

For more information, see About the Assured Replication Controls.

Directory Proxy Server

Replication assurance policies are not supported on PingDirectoryProxy. Replication client controls are passed through to the underlying PingDirectory server backend.

Schema Backend Replication

The schema backend is not supported by assured replication. Replication assurance policies that include criteria to match this backend are rejected.

Backward Compatibility

Server versions that support assured replication are backwards-compatible with prior versions that do not support assured replication.

WAN-Friendly Replication

Assured replication functions independently from WAN-Friendly Replication and the notion of WAN Gateways.

Global Configuration Properties

PingDirectory server provides two configurable global configuration properties that determine the timing of the assurance source and maximum number of replication backup updates recognized as an available source.

replication-assurance-source-timeout-suspend-duration: Specifies the amount of time a replication assurance source is suspended from assurance requirements if it experiences an assurance timeout. While suspended, the source is excluded from assurance requirements for all operations originating on this PingDirectory server. This avoids the situation of repeated timeouts caused by degraded or offline servers. The default time is 10 seconds.
replication-assurance-source-backlog-fast-start-threshold: Specifies the maximum number of replication backlog updates a replication assurance source can have and be immediately recognized as an available source. If a source connects to this PingDirectory server with more than the configured threshold backlog updates, it is excluded from assurance requirements for all operations originating from the server until it completes at least one assurance successfully, indicated by the server receiving an update acknowledgment message from it within the timeout window. The default maximum number of updates is 1000.

Configuring assured replication

About this task

It is common for all servers to have the same policy. The following example, which demonstrates the configuration of various assured replication policies, assumes that three servers are configured on localhost, on ports 1389, 2389, and 3389.

In this example, each server has a default Replication Assurance Policy with no assurance levels set.

Steps

To create request criteria for add operations on server 1, run dsconfig with the create-request-criteria option.

This request criteria is used to match any add operation with the Replication Assurance Policy that is configured in the following step.

Example:
```
$ bin/dsconfig create-request-criteria \
  --criteria-name Adds \
  --type simple \
  --set operation-type:add
```
To make all add operations assured with a level of processed-all-servers on server 1, set up the Replication Assurance Policy using dsconfig and specify the Adds request criteria configured in the previous step.

The processed-all-servers level indicates that all local servers in the topology must complete replay of the replication update before the response is sent to the client.

Example:
```
$ bin/dsconfig create-replication-assurance-policy \
  --policy-name "Adds Processed All Locally" \
  --set evaluation-order-index:1 \
  --set local-level:processed-all-servers \
  --set "timeout:500ms" \
  --set request-criteria:Adds
```

On server 1, repeat steps 1-2 for modify operations.

The Replication Assurance Policy Mods Received Any Locally ensures that at least one available local server must receive a replication modify update before a response is sent to the client.

Example:

$ bin/dsconfig create-request-criteria \
  --criteria-name Mods \
  --type simple \
  --set operation-type:modify

$ bin/dsconfig create-replication-assurance-policy \
  --policy-name "Mods Received Any Locally" \
  --set evaluation-order-index:2 \
  --set local-level:received-any-server \
  --set "timeout:500ms" \
  --set request-criteria:Mods

To set up the Adds and Mods request criteria and a Replication Assurance Policy on server 2, repeat steps 1-3.

Example:

$ bin/dsconfig create-request-criteria \
  --criteria-name Adds \
  --type simple \
  --set operation-type:add

$ bin/dsconfig create-request-criteria \
  --criteria-name Mods \
  --type simple \
  --set operation-type:modify

$ bin/dsconfig create-replication-assurance-policy \
  --policy-name "Adds Received Any Locally" \
  --set evaluation-order-index:1 \
  --set local-level:received-any-server \
  --set "timeout:500ms" \
  --set request-criteria:Adds

$ bin/dsconfig create-replication-assurance-policy \
  --policy-name "Mods Processed All Locally" \
  --set evaluation-order-index:2 \
  --set local-level:processed-all-servers \
  --set "timeout:500ms" \
  --set request-criteria:Mods

Leave server 3 with the default Replication Assurance Policy configured with no assurance levels or criteria. In practice, it is common for all servers to have the same assurance levels or criteria.

To list the policies on server 1 to confirm that they exist, run dsconfig with the list-replication-assurance-policies option.

Example:
```
$ bin/dsconfig list-replication-assurance-policies
```
Repeat step 5 for server 2 and server 3.

Server 3 should only show the Default Replication Assurance Policy.

To check the Replication Assurance counters on all servers before any add or modify operation, use ldapsearch.

The counters should be set to zero. These counters are on the replica server where the policy is matched and assigned.

Example:

For this example, on server 1, run the following command.

$ bin/ldapsearch --baseDN "cn=Replica dc_example_dc_com,cn=monitor" \
  "(objectclass=*)" | grep replication-assurance

replication-assurance-local-completed-normally: 0
replication-assurance-local-completed-abnormally: 0
replication-assurance-local-completed-with-timeout: 0
replication-assurance-local-completed-with-shutdown: 0
replication-assurance-local-completed-with-unavailable-server: 0
replication-assurance-remote-completed-normally: 0
replication-assurance-remote-completed-abnormally: 0
replication-assurance-remote-completed-with-timeout: 0
replication-assurance-remote-completed-with-shutdown: 0
replication-assurance-remote-completed-with-unavailable-server: 0

To check the Replication Summary table on all of the servers, use ldapsearch.

Example:

For this example, on server 1, run the following command.

$ bin/ldapsearch --baseDN "cn=Replication Summary dc_example_dc_com,cn=monitor" \
  "(objectclass=*)" | grep replication-assurance

replication-assurance-submitted-operations: 0
replication-assurance-local-completed-normally: 0
replication-assurance-local-completed-abnormally: 0
replication-assurance-local-completed-with-timeout: 0
replication-assurance-local-completed-with-shutdown: 0
replication-assurance-local-completed-with-unavailable-server: 0
replication-assurance-remote-completed-normally: 0
replication-assurance-remote-completed-abnormally: 0
replication-assurance-remote-completed-with-timeout: 0
replication-assurance-remote-completed-with-shutdown: 0
replication-assurance-remote-completed-with-unavailable-server: 0

To add an entry to server 1 on the PingDirectory server, use ldapmodify.

The counters should match the newly added entry to the Adds Processed All Locally policy and complete assured.

Example:

$ bin/ldapmodify --filename add-user.ldif --defaultAdd

$ bin/ldapsearch --baseDN "cn=Replica dc_example_dc_com,cn=monitor" \
  "(objectclass=)" | grep replication-assurance

replication-assurance-submitted-operations: 1
replication-assurance-local-completed-normally: 1
replication-assurance-local-completed-abnormally: 0
replication-assurance-local-completed-with-timeout: 0
replication-assurance-local-completed-with-shutdown: 0
replication-assurance-local-completed-with-unavailable-server: 0
replication-assurance-remote-completed-normally: 0
replication-assurance-remote-completed-abnormally: 0
replication-assurance-remote-completed-with-timeout: 0
replication-assurance-remote-completed-with-shutdown: 0
replication-assurance-remote-completed-with-unavailable-server: 0
replication-assurance-policy-matches: Adds Processed All Locally: 1
replication-assurance-policy-matches: Default Replication Assurance Policy: 0
replication-assurance-policy-matches: Mods Received Any Locally: 0
replication-assurance-local-level-uses: processed-all-servers: 1
replication-assurance-remote-level-uses: none: 1

$ bin/ldapsearch --baseDN "cn=Replication Summary dc_example_dc_com,cn=monitor" \
  "(objectclass=)" | grep replication-assurance

replication-assurance-submitted-operations: 1
replication-assurance-local-completed-normally: 1
replication-assurance-local-completed-abnormally: 0
replication-assurance-local-completed-with-timeout: 0
replication-assurance-local-completed-with-shutdown: 0
replication-assurance-local-completed-with-unavailable-server: 0
replication-assurance-remote-completed-normally: 0
replication-assurance-remote-completed-abnormally: 0
replication-assurance-remote-completed-with-timeout: 0
replication-assurance-remote-completed-with-shutdown: 0
replication-assurance-remote-completed-with-unavailable-server: 0

Perform a modify of an entry under dc=example,dc=com on server 1.

The counters should match the modify operation to the Mods Processed All Locally policy and the operations should complete assured.

Example:

$ bin/ldapsearch --baseDN "cn=Replica dc_example_dc_com,cn=monitor" \
  "(objectclass=)" | grep replication-assurance

replication-assurance-submitted-operations: 2
replication-assurance-local-completed-normally: 2
replication-assurance-local-completed-abnormally: 0
replication-assurance-local-completed-with-timeout: 0
replication-assurance-local-completed-with-shutdown: 0
replication-assurance-local-completed-with-unavailable-server: 0
replication-assurance-remote-completed-normally: 0
replication-assurance-remote-completed-abnormally: 0
replication-assurance-remote-completed-with-timeout: 0
replication-assurance-remote-completed-with-shutdown: 0
replication-assurance-remote-completed-with-unavailable-server: 0
replication-assurance-policy-matches: Adds Processed All Locally: 1
replication-assurance-policy-matches: Default Replication Assurance Policy: 0
replication-assurance-policy-matches: Mods Received Any Locally: 1
replication-assurance-local-level-uses: processed-all-servers: 1
replication-assurance-local-level-uses: received-any-server: 1
replication-assurance-remote-level-uses: none: 2

$ bin/ldapsearch --baseDN "cn=Replication Summary dc_example_dc_com,cn=monitor" \
  "(objectclass=)" | grep replication-assurance

replication-assurance-submitted-operations: 2
replication-assurance-local-completed-normally: 2
replication-assurance-local-completed-abnormally: 0
replication-assurance-local-completed-with-timeout: 0
replication-assurance-local-completed-with-shutdown: 0
replication-assurance-local-completed-with-unavailable-server: 0
replication-assurance-remote-completed-normally: 0
replication-assurance-remote-completed-abnormally: 0
replication-assurance-remote-completed-with-timeout: 0
replication-assurance-remote-completed-with-shutdown: 0
replication-assurance-remote-completed-with-unavailable-server: 0

Result:

You have successfully configured Assured Replication.

About the assured replication controls

The LDAP SDK for Java provides an implementation of an LDAP control that can be included in add, bind, modify, modifyDN, and certain extended requests to indicate the optimal level of replication assurance for the associated operation. The OID for this control is 1.3.6.1.4.1.30221.2.5.28 and can have a criticality of either TRUE or FALSE.

For more information, see the LDAP SDK Javadoc for the AssuredReplicationRequestControl class.