PingAuthorize

The server is unresponsive

If the server process is running and appears to be accepting connections, but doesn’t respond to requests received on those connections, first check the current server state by using the bin/server-state command.

If it appears that the problem is with the server software or the JVM in which it is running, then you need to work with your authorized support provider to fully diagnose the problem and determine the best course of action to correct it.

If the server is unresponsive, then potential reasons for this behavior include:

  • If all worker threads are busy processing other client requests, then new requests that arrive will be forced to wait in the work queue until a worker thread becomes available. If this is the case, then a stack trace obtained using the jstack command shows that all of the worker threads are busy and none of them are waiting for new requests to process.

    A dedicated thread pool can be used for processing administrative operations. This thread pool enables diagnosis and corrective action if all other worker threads are processing operations. To request that operations use the administrative thread pool, using the ldapsearch command for example, include the --useAdministrativeSession option. The requester must have the use-admin-session privilege (included for root users). By default, eight threads are available for this purpose. This can be changed with the num-administrative-session-worker-threads property in the work queue configuration.

    If all of the worker threads are tied up processing the same operation for a long time, the server will also issue an alert that it might be deadlocked, which might not actually be the case. All threads might be tied up processing unindexed searches.

  • If a request handler is stuck performing some expensive processing for a client connection, then other requests sent to the server on connections associated with that request handler are forced to wait until the request handler is able to read data on those connections. If this is the case, then only some of the connections can experience this behavior (unless there is only a single request handler, in which it will impact all connections), and stack traces obtained using the jstack command show that a request handler thread is continuously blocked rather than waiting for new requests to arrive. Note that this scenario is a theoretical problem and one that has not appeared in production.

  • If the JVM in which the server is running is not properly configured, then it can be forced to spend a significant length of time performing garbage collection, and in severe cases, could cause significant interruptions in the execution of Java code. In such cases, a stack trace obtained from a pstack of the native process should show that most threads are idle, but at least one thread performing garbage collection is active. It is also likely that one or a small number of CPUs are 100% busy while all other CPUs are mostly idle. The server will also issue an alert after detecting a long JVM pause (due to garbage collection). The alert will include details of the pause.

  • If the JVM in which the server is running has hung for some reason, then the pstack utility should show that one or more threads are blocked and unable to make progress. In such cases, the system CPUs should be mostly idle.

  • If a network or firewall configuration problem arises, then attempts to communicate with the server cannot be received by the server. In that case, a network sniffer like snoop or tcpdump should show that packets sent to the system on which the server is running are not receiving TCP acknowledgement.

  • If the system on which the server is running has become hung or lost power with a graceful shutdown, then the behavior is often similar to that of a network or firewall configuration problem.