If nothing else helps, then you might need to resort to low-level debugging options.
In this case, the best option is probably to enable the Java Virtual Machine
(JVM’s) support for TLS debugging. Many of the command-line tools provided with the
Directory Server, like ldapsearch, offer an
argument that can simplify this process, but for other tools, you can edit the
config/java.properties file to add
-Djavax.net.debug=all to the set of properties for the desired tool
and then run the
bin/dsjavaproperties command to make the change take
effect. The next time you run the tool, you will get voluminous output detailing all of
the TLS-related processing that the JVM is performing. Something in that output should
allow you or our support personnel to identify the issue.
This low-level debugging is also available within the server by adding
-Djavax.net.debug=all to the
property in the config/java.properties file, running
bin/dsjavaproperties, and restarting the server. Because this
requires a server restart (and then another one when you want to turn off the
debugging), it is not a very attractive option. You might be able to obtain more
information about the problem without a restart by using the debugging support built
into the server. You can do that with the following configuration changes.
dsconfig create-debug-target \ --publisher-name "File-Based Debug Logger" \ --target-name com.unboundid.directory.server.extensions.TLSConnectionSecurityProvider \ --set debug-level:verbose dsconfig set-log-publisher-prop \ --publisher-name "File-Based Debug Logger" \ --set enabled:true
After these changes, the logs/debug file captures a substantial amount of information about the TLS-related processing that the server is performing. It won’t have quite as much detail as the debugging information built into the JVM, but it can still allow pinpointing the cause of the problem and identifying the solution. When this debugging is no longer needed, the debug log publisher can be disabled, and the debug target can be removed.
dsconfig set-log-publisher-prop \ --publisher-name "File-Based Debug Logger" \ --set enabled:false dsconfig delete-debug-target \ --publisher-name "File-Based Debug Logger" \ --target-name com.unboundid.directory.server.extensions.TLSConnectionSecurityProvider
If you need to troubleshoot TLS communication with a non-Java client that doesn’t offer its own TLS debugging mechanism, and if the server-side debugging support is not sufficient, then the best remaining option can be to use a network protocol analyzer to capture the communication between the client and the server and examine its content. The free and open source Wireshark utility, available at https://www.wireshark.org/, is an excellent graphical tool that runs on a variety of platforms and provides excellent support for understanding TLS communication. Even if you’re unable to decipher the encrypted content, you can see at least some of the handshake messages. Unfortunately, when using TLS version 1.3, more of the handshake is now encrypted than with earlier versions of the protocol. This is terrific for security and privacy but unfortunately can interfere with troubleshooting attempts.