PingDirectory

Repairing broken listener certificate trust in replication

PingDirectory replication configuration depends on listener certificates. When certificate trust is broken between servers, replication and client traffic continues, but topology changes are disrupted.

PingDirectory uses a mirrored subtree service to keep configuration subtrees synchronized across a replicated topology. This synchronization depends on listener certificates being trusted between servers. If you discover (for example, during a topology change or from a peer connection alert) that these certificates are out of sync, the certificate trust is broken and the topology can’t be properly mirrored. For example, when one server tries to send changes to trusted servers, the operation fails if the receiving servers have a different master than the sending server, or don’t have a master at all.

Common causes of a broken mirrored subtree:

  • Replacing one or more LDAPS connection handler certificates before adding the listener certificate trust to the topology

  • Updating the key store on disk without running the replace-certificate tool

  • Replacing the listener certificate trust with the new certificate without including the old certificate

  • Using an older or untested custom script to replace the certificate

  • Making a typo when replacing the certificate

  • Allowing the server certificate to expire

When you have a large number of servers in your environment, and your listener certificates fall out of sync, repairing trust and restoring topology mirroring can become complex. Rollback after a certificate rotation might not be possible because the new certificate is already serving client traffic. To troubleshoot, you check the logs and statuses for multiple servers. After you find the root cause, you update all server members lacking the appropriate trust, but you have to force each of these servers as master because the topology is read-only, due to the lack of trust.

Instead of following this long and complicated troubleshooting procedure, you can use the repair-topology-listener-certificates CLI tool. This tool helps restore trust and bring replicated environments back to a working state, greatly reducing the effort required to fix this issue. Learn more about using the repair-topology-listener-certificates tool.

Using the repair-topology-listener-certificates tool

Before you begin

Forced masters

The repair-topology-listener-certificates tool can fail if there are multiple masters in the topology. You must ensure no servers have force-as-master-for-mirrored-data set to true in the global configuration of any server in the topology.

Topology credentials

To update all server instances, you must supply an ID that exists on all servers in the topology and has the same password. If your root user accounts have differing passwords on all server nodes, you can use the topology administrator credentials, which consist of the same ID and password across the cluster.

About this task

You can use the repair-topology-listener-certificates tool to repair a broken PingDirectory topology by replacing the listener certificates for one or more server instances. The tool connects to each server in the topology and collects the certificates that they present during TLS negotiation for LDAPS, LDAP StartTLS, and HTTPS. It then examines the topology registry for each server to identify any updates that need to be made and attempts to apply those updates.

Because you might need to run the tool multiple times to repair trust across the entire topology, each execution generates log files and LDIF change files to a separate directory, so that you can correlate changes to a specific run. By default, these subdirectories are stored within the current working directory and are named as follows: repair-topology-listener-certificates-<timestamp>.

Important arguments for repair-topology-listener-certificates are listed in the following tables. See the CLI for a complete list.

LDAP connection and authentication arguments
Argument Description

--hostname <host>

The IP address or resolvable name to use to connect to a specific instance of the PingDirectory server. If you don’t provide this value, the tool uses localhost.

--port <port>

The port to use to connect to the server. If you don’t provide this value, the tool uses a default value of 389.

--bindDN <DN>

The DN to use to bind to the server when performing simple authentication.

--bindPassword <password>

The password to use to bind to the server when performing simple authentication or a password-based SASL mechanism.

--bindPasswordFile <path>

The path to the file containing the password to use to bind to the server when performing simple authentication or a password-based SASL mechanism.

--promptForBindPassword

Indicates that the tool should interactively prompt the user for the bind password.

--useSSL

The tool uses SSL when communicating with the server.

--useStartTLS

The tool uses StartTLS when communicating with the server.

--defaultTrust

The tool uses the JVM’s default trust store, and optionally an additional trust store specified using the --trustStorePath argument, to non-interactively determine whether to trust any certificate chain presented during TLS negotiation. If the chain can’t be trusted based on any of those sources, then negotiation fails without prompting the user.

--trustAll

The tool trusts any certificate presented by the server.

--keyStorePath <path>

The path to the file to use as the key store for obtaining client certificates when communicating securely with the server.

--keyStorePassword <password>

The password to use to access the key store contents.

--keyStorePasswordFile <path>

The path to the file containing the password to use to access the key store contents.

--promptForKeyStorePassword

Indicates that the tool should interactively prompt the user for the password to use to access the key store contents.

--keyStoreFormat <format>

The format for the key store file. For example, JKS, PKCS12, PKCS11, or BCFKS.

--trustStorePath <path>

The path to the file to use as trust store when determining whether to trust a certificate presented by the server.

--trustStorePassword <password>

The password to use to access the trust store contents.

--trustStorePasswordFile <path>

The path to the file containing the password to use to access the trust store contents.

--promptForTrustStorePassword

Indicates that the tool should interactively prompt the user for the password to use to access the trust store contents.

--trustStoreFormat <format>

The format for the trust store file. For example: JKS, PKCS12, PKCS11, or BCFKS.

--verifyCertificateHostnames

Indicates that the tool should verify that the host name or IP address used to establish connections to the server matches an address for which the server’s TLS certificate was issued.

--certNickname <nickname>

The alias of the client certificate in the key store to present to the server for SSL client authentication.

--enableSSLDebugging

Enables Java’s low-level support for debugging SSL/TLS communication. This is equivalent to setting the javax.net.debug property to all.

--saslOption <name=value>

A name-value pair that provides information for the server to use when performing SASL authentication.

--useSASLExternal

The tool uses the SASL EXTERNAL mechanism to authenticate.

--helpSASL

Provides information about the supported SASL mechanisms, including the properties available for use with each.
Usage arguments
Argument Description

--dryRun

The tool identifies any updates that need to be applied to the topology registry but doesn’t attempt to apply them. The identified changes are written as LDIF files in the output directory.

--forceAsMasterDelay <delay>

The length of time that the tool should wait to update topology data for a server instance after forcing that instance to temporarily act as the topology master.

If you include this argument, you must supply a value that includes an integer followed by a time unit. For no delay, supply a value of 0 seconds. If you don’t include this argument, the tool uses a default delay of 10 seconds.

--interactive

Launches the tool in interactive mode.

--outputDirectory <path>

The path to a directory where the tool should write output files, including a log file with detailed information about the processing performed by this tool and LDIF files with any changes to be applied.

If you don’t supply this argument, the tool creates a subdirectory named repair-topology-listener-certificates-<timestamp> in the current working directory.

Steps

  1. Optional: To review the changes needed before applying them, run the tool using the --dryRun argument.

    Example:

    $ bin/repair-topology-listener-certificates \
  --hostname ds1.example.com \
  --port 636 --useSSL \
  --trustStorePath /path/to/ds1/trust/store \
  --bindDN "cn=Directory Manager" \
  --bindPasswordFile /path/to/password.txt \
  --dryRun

  2. Run the tool to apply the necessary topology registry changes.

    Choose from:

    • Remove the --dryRun argument and re-run the command to apply the changes identified during the dry run.

    • Build a command using the arguments needed from the previous tables and run the command to identify and apply changes.

    • Run repair-topology-listener-certificates --interactive to enter interactive mode and enable the CLI to prompt you for any information the tool needs to run.

  3. Review the tool output and verify whether the topology trust has been restored:

    If trust is fully restored, no further steps are needed.

    Troubleshooting:

    If trust hasn’t been fully restored:

    1. Review the command and make any adjustments needed.

    2. Re-run the command.

    3. Return to the beginning of this step.