PingDirectory

PingDirectory server troubleshooting information

The PingDirectory server has a comprehensive default set of log files and monitor entries that are useful when troubleshooting a particular server problem.

Error log

By default, this log file is available at logs/errors below the server install root and it provides information about warnings, errors, and other significant events that occur within the server. A number of messages are written to this file on startup and shutdown, but while the server is running there is normally little information written to it. In the event that a problem does occur, however, the server writes information about that problem to this file.

The following is an example of a message that might be written to the error log:

[11/Apr/2011:10:31:53.783 -0500] category=CORE severity=NOTICE msgID=458887 msg="The Directory Server has started successfully"

The category field provides information about the area of the server from which the message was generated. Available categories include ACCESS_CONTROL, ADMIN, ADMIN_TOOL, BACKEND, CONFIG, CORE, DSCONFIG, EXTENSIONS, PROTOCOL, SCHEMA, JEB, SYNC, LOG, PLUGIN, PROXY, QUICKSETUP, REPLICATION, RUNTIME_INFORMATION, TASK, THIRD_PARTY, TOOLS, USER_DEFINED, UTIL, and VERSION.

The severity field provides information about how severe the server considers the problem to be. Available severities include:

  • DEBUG – Used for messages that provide verbose debugging information and do not indicate any kind of problem. Note that this severity level is rarely used for error logging, because the server provides a separate debug logging facility.

  • INFORMATION – Used for informational messages that can be useful from time to time but are not normally something that administrators need to see.

  • MILD_WARNING – Used for problems that the server detects, which can indicate something unusual occurred, but the warning does not prevent the server from completing the task it was working on. These warnings are not normally something that should be of concern to administrators.

  • MILD_ERROR – Used for problems detected by the server that prevented it from completing some processing normally but that are not considered to be a significant problem requiring administrative action.

  • NOTICE – Used for information messages about significant events that occur within the server and are considered important enough to warrant making available to administrators under normal conditions.

  • SEVERE_WARNING – Used for problems that the server detects that might lead to bigger problems in the future and should be addressed by administrators.

  • SEVERE_ERROR – Used for significant problems that have prevented the server from successfully completing processing and are considered important.

  • FATAL_ERROR – Used for critical problems that arise which might leave the server unable to continue processing operations normally.

The messages written to the error log can be filtered based on their severities in two ways. First, the error log publisher has a default-severity property, which can be used to specify the severity of messages logged regardless of their category. By default, this includes the NOTICE, SEVERE_WARNING, SEVERE_ERROR, and FATAL_ERROR severities.

You can override these severities on a per-category basis using the override-severity property. If this property is used, then each value should consist of a category name followed by an equal sign and a comma-delimited set of severities that should be logged for messages in that category. For example, the following override severity would enable logging at all severity levels in the PROTOCOL category:

protocol=debug,information,mild-warning,mild-error,notice,severe-warning,severe-error,fatal-error

Note that for the purposes of this configuration property, any underscores in category or severity names should be replaced with dashes. Also, severities are not inherently hierarchical, so enabling the DEBUG severity for a category will not automatically enable logging at the INFORMATION, MILD_WARNING, or MILD_ERROR severities.

The error log configuration can be altered on the fly using tools like dsconfig, the Administrative Console, or the LDIF connection handler, and changes will take effect immediately. You can configure multiple error logs that are active in the server at the same time, writing to different log files with different configurations. For example, a new error logger can be activated with a different set of default severities to debug a short-term problem, and then that logger can be removed after the problem is resolved, so that the normal error log does not contain any of the more verbose information.

server.out log

The server.out file holds any information written to standard output or standard error while the server is running. Normally, it includes a number of messages written at startup and shutdown, as well as information about any administrative alerts generated while the server is running. In most cases, this information is also written to the error log. The server.out file can also contain output generated by the JVM. For example, if garbage collection debugging is enabled, or if a stack trace is requested via "kill -QUIT" as described in a later section, then output is written to this file.

Debug log

The debug log provides a means of obtaining information that can be used for troubleshooting problems but is not necessary or desirable to have available while the server is functioning normally. As a result, the debug log is disabled by default, but it can be enabled and configured at any time.

Some of the most notable configuration properties for the debug log publisher include:

  • enabled – Indicates whether debug logging is enabled. By default, it is disabled.

  • log-file – Specifies the path to the file to be written. By default, debug messages are written to the logs/debug file.

  • default-debug-level – Specifies the minimum log level for debug messages that should be written. The default value is “error,” which only provides information about errors that occur during processing (for example, exception stack traces). Other supported debug levels include warning, info, and verbose. Note that unlike error log severities, the debug log levels are hierarchical. Configuring a specified debug level enables any debugging at any higher levels. For example, configuring the info debug level automatically enables the warning and error levels.

  • default-debug-category – Specifies the categories for debug messages that should be written. Some of the most useful categories include caught (provides information and stack traces for any exceptions caught during processing), database-access (provides information about operations performed in the underlying database), protocol (provides information about ASN.1 and LDAP communication performed by the server), and data (provides information about raw data read from or written to clients).

As with the error and access logs, multiple debug loggers can be active in the server at any time with different configurations and log files to help isolate information that might be relevant to a particular problem. NOTE: Enabling one or more debug loggers can have a significant impact on server performance. We recommend that debug loggers be enabled only when necessary, and then be scoped so that only pertinent debug information is recorded.

Debug targets can be used to further pare down the set of messages generated. For example, you can specify that the debug logs be generated only within a specific class or package. If you need to enable the debug logger, you should work with your authorized support provider to best configure the debug target and interpret the output.

Replication repair log

The replication repair log is written to logs/replication by default and records information about processing performed by the replication repair service. This log is used to resolve replication conflicts that can arise. For example, if the same entry is modified at the same time on two different systems, or if an attempt is made to create entries with the same DN at the same time on two different systems, the PingDirectory server records these events.

Config audit log and the configuration archive

The configuration audit log provides a record of any changes made to the server configuration while the server is online. This information is written to the logs/config-audit.log file and provides information about the configuration change in the form that can be used to perform the operation in a non-interactive manner with the dsconfig command. Other information written for each change includes:

  • Time that the configuration change was made.

  • Connection ID and operation ID for the corresponding change, which can be used to correlate it with information in the access log.

  • DN of the user requesting the configuration change and the method by which that user authenticated to the server.

  • Source and destination addresses of the client connection.

  • Command that can be used to undo the change and revert to the previous configuration for the associated configuration object.

In addition to information about the individual changes that are made to the configuration, the server maintains complete copies of all previous configurations. These configurations are provided in the config/archived-configs directory and are gzip-compressed copies of the config/config.ldif file in use before the configuration change was made. The file names contain timestamps that indicate when that configuration was first used.

Access and audit log

The access log provides information about operations processed within the server. The default access log file is written to logs/access, but multiple access loggers can be active at the same time, each writing to different log files and using different configurations.

By default, a single access log message is generated, which combines the elements of request, forward, and result messages. If an error is encountered while attempting to process the request, then one or more forward-failed messages can also be generated.

[01/Jun/2011:11:10:19.692 -0500] CONNECT conn=49 from="127.0.0.1" to="127.0.0.1"
  protocol="LDAP+TLS" clientConnectionPolicy="default"
[01/Jun/2011:11:10:19.764 -0500] BIND RESULT conn=49 op=0 msgID=1 version="3"
  dn="cn=Directory Manager" authType="SIMPLE" resultCode=0 etime=0.401
  authDN="cn=Directory Manager,cn=Root DNs,cn=config" clientConnectionPolicy="default"
[01/Jun/2011:11:10:19.769 -0500] SEARCH RESULT conn=49 op=1 msgID=2
  base="ou=People,dc=example,dc=com" scope=2 filter="(uid=1)" attrs="ALL"
  resultCode=0 etime=0.549 entriesReturned=1
[01/Jun/2011:11:10:19.788 -0500] DISCONNECT conn=49 reason="Client Unbind"

Each log message includes a timestamp indicating when it was written, followed by the operation type, the connection ID (which is used for all operations processed on the same client connection), the operation ID (which can be used to correlate the request and response log messages for the operation), and the message ID used in LDAP messages for this operation.

The remaining content for access log messages varies based on the type of operation being processed, and whether it is a request or a result message. Request messages generally include the most pertinent information from the request, but generally omit information that is sensitive or not useful.

Result messages include a resultCode element that indicates whether the operation was successful or if failed and an etime element that indicates the length of time in milliseconds that the server spent processing the operation. Other elements that might be present include the following:

  • origin=replication – Operation that was processed as a result of data synchronization (for example, replication) rather than a request received directly from a client.

  • message – Text that was included in the diagnosticMessage field of the response sent to the client.

  • additionalInfo – Additional information about the operation that was not included in the response sent back to the client.

  • authDN – DN of the user that authenticated to the server (typically only included in bind result messages).

  • authzDN – DN of an alternate authorization identify used when processing the operation (for example, if the proxied authorization control was included in the request).

  • authFailureID – Unique identifier associated with the authentication failure reason (only included in non-successful bind result messages).

  • authFailureReason – Information about the reason that a bind operation failed that might be useful to administrators but was not included in the response to the client for security reasons.

  • responseOID – OID included in an extended response returned to the client.

  • entriesReturned – Number of matching entries returned to the client for a search operation.

  • unindexed=true – Indicates that the associated search operation could not be sufficiently processed using server indexes and a significant traversal through the database was required.

Note that this is not an exhaustive list, and elements that are not listed here can also be present in access log messages. The LDAP SDK for Java provides an API for parsing access log messages and provides access to all elements that they can contain.

The server provides a second access log implementation called the audit log, which is used to provide detailed information about write operations (add, delete, modify, and modify DN) processed within the server. If the audit log is enabled, the entire content of the change is written to the audit log file (which defaults to logs/audit) in LDIF form.

The server also provides a very rich classification system that can be used to filter the content for access log files. This can be helpful when debugging problems with client applications, because it can restrict log information to operations processed only by a particular application (for example, based on IP address and/or authentication DN), only failed operations, or only operations taking a long time to complete, etc.

Setup log

The setup tool writes a log file providing information about the processing that it performs. By default, this log file is written to logs/setup.log although a different name may be used if a file with that name already exists, because the setup tool has already been run. The full path to the setup log file is provided when the setup tool has completed.

Tool log

Many of the administrative tools provided with the server (for example, import-ldif, export-ldif, backup, restore, etc.) can take a significant length of time to complete write information to standard output or standard error or both while the tool is running. They also write additional output to files in the logs/tools directory (for example, logs/tools/ import-ldif.log). The information written to these log files can be useful for diagnosing problems encountered while they were running. When running using the server tasks interface, log messages generated while the task is running can alternately be written to the server error log file.

je.info and je.config files

The primary datastore used by the PingDirectory server is the Oracle Berkeley DB Java Edition (JE). The PingDirectory server provides two primary sources of information about processing within the database.The first is logging performed by the JE code itself, and is written into the

je.info.0 file in the server containing the database files (for example, db/userRoot/je.info.0). In the event of a problem within JE itself, useful information about the nature of the problem may be written to this log. The level of information written to this log file is controlled by the db-logging-level property in the backend configuration object. It uses the standard Java logging framework for logging messages, so the standard SEVERE, WARNING, INFO, CONFIG, FINE, FINER, and FINEST levels are available.

The second is configuration information used when opening the database environment. When the backend database environment is opened, then the PingDirectory server will also write a file named je.config in the server containing the database files (for example, db/userRoot/je.config) with information about the configuration used.

LDAP SDK debug log

This log can be used to help examine the communication between the PingDirectory server and the PingDirectoryProxy server. It contains information about exceptions that occur during processing, problems establishing and terminating network connections, and problems that occur during the reading and writing of LDAP messages and LDIF entries. You can configure the types of debugging that should be enabled, the debug level that should be used, and whether debug messages should include stack traces. As for other file-based loggers, you can also specify the rotation and retention policies.