Check the current server state by using the bin/server-state command. If the server process is running and appears to be accepting connections but does not respond to requests received on those connections, potential reasons for this include:

  • If all worker threads are busy processing other client requests, new requests are forced to wait until a worker thread becomes available. A stack trace can be obtained using the jstack command to show the state of the worker threads and the waiting requests.

    If all worker threads are processing the same requests for a long time, the server sends an alert that it might be deadlocked. All threads might be tied up processing unindexed searches.

  • If a request handler is busy with a client connection, other requests sent through that request handler are forced to wait until it is able to read data. If there is only one request handler, all connections are impacted. Stack traces obtained using the jstack command will show that a request handler thread is continuously blocked.
  • If the Java Virtual Machine (JVM) in which the server is running is not properly configured, it can spend too much time performing garbage collection. The effect on the server is similar to that of a network or firewall configuration problem. A stack trace obtained with the pstack utility will show that most threads are idle except the one performing garbage collection. It is also likely that a small number of CPUs is 100% busy while all other CPUs are idle. The server will also issue an alert after detecting a long JVM pause that will include details.
  • If the JVM in which the server is running has hung, the pstack utility should show that one or more threads are blocked and unable to make progress. In such cases, the system CPUs should be mostly idle.
  • If a there is a network or firewall configuration problem, communication attempts with the server will fail. A network sniffer will show that packets sent to the system are not receiving TCP acknowledgment.
  • If the host system is hung or lost power with a graceful shutdown, the server will be unresponsive.