--- title: Repairing broken listener certificate trust in replication description: PingDirectory replication configuration depends on listener certificates. When certificate trust is broken between servers, replication and client traffic continues, but topology changes are disrupted. component: pingdirectory version: 11.0 page_id: pingdirectory:managing_servers_and_certificates:pd_ds_repair_broken_trust_replication canonical_url: https://docs.pingidentity.com/pingdirectory/11.0/managing_servers_and_certificates/pd_ds_repair_broken_trust_replication.html revdate: June 14, 2024 page_aliases: ["pd_ds_repair_topology_listener_certificates.adoc"] section_ids: using-the-repair-topology-listener-certificates-tool: Using the repair-topology-listener-certificates tool before-you-begin: Before you begin about-this-task: About this task steps: Steps example: Example: choose-from: Choose from: troubleshooting: Troubleshooting: --- # Repairing broken listener certificate trust in replication PingDirectory replication configuration depends on listener certificates. When certificate trust is broken between servers, replication and client traffic continues, but topology changes are disrupted. PingDirectory uses a mirrored subtree service to keep configuration subtrees synchronized across a replicated topology. This synchronization depends on listener certificates being trusted between servers. If you discover (for example, during a topology change or from a peer connection alert) that these certificates are out of sync, the certificate trust is broken and the topology can't be properly mirrored. For example, when one server tries to send changes to trusted servers, the operation fails if the receiving servers have a different master than the sending server, or don't have a master at all. | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Common causes of a broken mirrored subtree:- Replacing one or more LDAPS connection handler certificates before adding the listener certificate trust to the topology - Updating the key store on disk without running the `replace-certificate` tool - Replacing the listener certificate trust with the new certificate without including the old certificate - Using an older or untested custom script to replace the certificate - Making a typo when replacing the certificate - Allowing the server certificate to expire | When you have a large number of servers in your environment, and your listener certificates fall out of sync, repairing trust and restoring topology mirroring can become complex. Rollback after a certificate rotation might not be possible because the new certificate is already serving client traffic. To troubleshoot, you check the logs and statuses for multiple servers. After you find the root cause, you update all server members lacking the appropriate trust, but you have to force each of these servers as master because the topology is read-only, due to the lack of trust. Instead of following this long and complicated troubleshooting procedure, you can use the `repair-topology-listener-certificates` CLI tool. This tool helps restore trust and bring replicated environments back to a working state, greatly reducing the effort required to fix this issue. ## Using the `repair-topology-listener-certificates` tool ### Before you begin * Forced masters The `repair-topology-listener-certificates` tool can fail if there are multiple masters in the topology. You must ensure no servers have `force-as-master-for-mirrored-data` set to `true` in the global configuration of any server in the topology. * Topology credentials To update all server instances, you must supply an ID that exists on all servers in the topology and has the same password. If your root user accounts have differing passwords on all server nodes, you can use the topology administrator credentials, which consist of the same ID and password across the cluster. ### About this task You can use the `repair-topology-listener-certificates` tool to repair a broken PingDirectory topology by replacing the listener certificates for one or more server instances. The tool connects to each server in the topology and collects the certificates that they present during TLS negotiation for LDAPS, LDAP StartTLS, and HTTPS. It then examines the topology registry for each server to identify any updates that need to be made and attempts to apply those updates. | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Because you might need to run the tool multiple times to repair trust across the entire topology, each execution generates log files and LDIF change files to a separate directory, so that you can correlate changes to a specific run. By default, these subdirectories are stored within the current working directory and are named as follows: `repair-topology-listener-certificates-`. | Important arguments for `repair-topology-listener-certificates` are listed in the following tables. See the CLI for a complete list. **LDAP connection and authentication arguments** | Argument | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--hostname ` | The IP address or resolvable name to use to connect to a specific instance of the PingDirectory server. If you don't provide this value, the tool uses `localhost`. | | `--port ` | The port to use to connect to the server. If you don't provide this value, the tool uses a default value of `389`. | | `--bindDN ` | The DN to use to bind to the server when performing simple authentication. | | `--bindPassword ` | The password to use to bind to the server when performing simple authentication or a password-based SASL mechanism. | | `--bindPasswordFile ` | The path to the file containing the password to use to bind to the server when performing simple authentication or a password-based SASL mechanism. | | `--promptForBindPassword` | Indicates that the tool should interactively prompt the user for the bind password. | | `--useSSL` | The tool uses SSL when communicating with the server. | | `--useStartTLS` | The tool uses StartTLS when communicating with the server. | | `--defaultTrust` | The tool uses the JVM's default trust store, and optionally an additional trust store specified using the `--trustStorePath` argument, to non-interactively determine whether to trust any certificate chain presented during TLS negotiation. If the chain can't be trusted based on any of those sources, then negotiation fails without prompting the user. | | `--trustAll` | The tool trusts any certificate presented by the server. | | `--keyStorePath ` | The path to the file to use as the key store for obtaining client certificates when communicating securely with the server. | | `--keyStorePassword ` | The password to use to access the key store contents. | | `--keyStorePasswordFile ` | The path to the file containing the password to use to access the key store contents. | | `--promptForKeyStorePassword` | Indicates that the tool should interactively prompt the user for the password to use to access the key store contents. | | `--keyStoreFormat ` | The format for the key store file. For example, JKS, PKCS12, PKCS11, or BCFKS. | | `--trustStorePath ` | The path to the file to use as trust store when determining whether to trust a certificate presented by the server. | | `--trustStorePassword ` | The password to use to access the trust store contents. | | `--trustStorePasswordFile ` | The path to the file containing the password to use to access the trust store contents. | | `--promptForTrustStorePassword` | Indicates that the tool should interactively prompt the user for the password to use to access the trust store contents. | | `--trustStoreFormat ` | The format for the trust store file. For example: JKS, PKCS12, PKCS11, or BCFKS. | | `--verifyCertificateHostnames` | Indicates that the tool should verify that the host name or IP address used to establish connections to the server matches an address for which the server's TLS certificate was issued. | | `--certNickname ` | The alias of the client certificate in the key store to present to the server for SSL client authentication. | | `--enableSSLDebugging` | Enables Java's low-level support for debugging SSL/TLS communication. This is equivalent to setting the `javax.net.debug` property to `all`. | | `--saslOption ` | A name-value pair that provides information for the server to use when performing SASL authentication. | | `--useSASLExternal` | The tool uses the SASL EXTERNAL mechanism to authenticate. | | `--helpSASL` | Provides information about the supported SASL mechanisms, including the properties available for use with each. | **Usage arguments** | Argument | Description | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `--dryRun` | The tool identifies any updates that need to be applied to the topology registry but doesn't attempt to apply them. The identified changes are written as LDIF files in the output directory. | | `--forceAsMasterDelay ` | The length of time that the tool should wait to update topology data for a server instance after forcing that instance to temporarily act as the topology master.If you include this argument, you must supply a value that includes an integer followed by a time unit. For no delay, supply a value of `0 seconds`. If you don't include this argument, the tool uses a default delay of 10 seconds. | | `--interactive` | Launches the tool in interactive mode. | | `--outputDirectory ` | The path to a directory where the tool should write output files, including a log file with detailed information about the processing performed by this tool and LDIF files with any changes to be applied.If you don't supply this argument, the tool creates a subdirectory named `repair-topology-listener-certificates-` in the current working directory. | ### Steps 1. (Optional) To review the changes needed before applying them, run the tool using the `--dryRun` argument. #### Example: ```shell $ bin/repair-topology-listener-certificates \ --hostname ds1.example.com \ --port 636 --useSSL \ --trustStorePath /path/to/ds1/trust/store \ --bindDN "cn=Directory Manager" \ --bindPasswordFile /path/to/password.txt \ --dryRun ``` 2. Run the tool to apply the necessary topology registry changes. #### Choose from: * Remove the `--dryRun` argument and re-run the command to apply the changes identified during the dry run. * Build a command using the arguments needed from the previous tables and run the command to identify and apply changes. * Run `repair-topology-listener-certificates --interactive` to enter interactive mode and enable the CLI to prompt you for any information the tool needs to run. 3. Review the tool output and verify whether the topology trust has been restored: If trust is fully restored, no further steps are needed. #### Troubleshooting: If trust hasn't been fully restored: 1. Review the command and make any adjustments needed. 2. Re-run the command. 3. Return to the beginning of this step.