---
title: HTTP-based monitoring
description: DS servers publish monitoring information at these HTTP endpoints:
component: pingds
version: 7.2
page_id: pingds:monitoring-guide:http-monitoring
canonical_url: https://docs.pingidentity.com/pingds/8.1/monitoring-guide/http-monitoring.html
llms_txt: https://docs.pingidentity.com/pingds/llms.txt
docs_for_agents: https://developer.pingidentity.com/build-with-ai/docs-for-agents.md
revdate: 2022-11-21T14:12:55Z
superseded_by: https://docs.pingidentity.com/pingds/8.1/monitoring-guide/http-monitoring.html
section_ids:
  monitoring-liveness-http: Server is alive (HTTP)
  monitoring-health-http: Server health (HTTP)
  monitoring-health-status-prometheus: Server health (Prometheus)
  monitoring-replication-delay-http: Replication delay (Prometheus)
  monitoring-disk-space: Disk space (Prometheus)
  monitoring-certificate-expiration: Certificate expiration (Prometheus)
  monitoring-operation-stats-http: Request statistics (Prometheus)
  monitoring-work-queue-http: Work queue (Prometheus)
  monitoring-entry-counts-http: Database size (Prometheus)
  monitoring-activity-http: Active users (Prometheus)
  monitoring-index-reads-http: Index use (Prometheus)
  monitoring-filters-http: Filtering results (Prometheus)
---

# HTTP-based monitoring

DS servers publish monitoring information at these HTTP endpoints:

* `/alive`

  Whether the server is currently *alive*, meaning that its internal checks have not found any errors that would require administrative action.

* `/healthy`

  Whether the server is currently *healthy*, meaning that it is alive, that the replication server is accepting connections on the configured port, and that any replication delays are below the configured threshold.

* `/metrics/api`

  Read-only access through [Forgerock Common REST](../rest-guide/rest-operations.html) to a JSON-based view of `cn=monitor` and the monitoring backend.

  The endpoint represents a collection where each LDAP entry maps to a resource under `/metrics/api`. Use a REST [Query](../rest-guide/query-rest.html) with a `_queryFilter` parameter to access this endpoint. To return all resources, use `/metrics/api?_queryFilter=true`.

* `/metrics/prometheus`

  Monitoring information for [Prometheus monitoring software](https://prometheus.io/).

  For details, see [Prometheus metrics reference](monitoring-metrics-prometheus.html).

  The following example command accesses the Prometheus endpoint:

  ```bash
  $ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus
  ```

To give a regular user privileges to read monitoring data, see [Monitor privilege](ldap-monitoring.html#monitoring-read-privilege).

## Server is alive (HTTP)

The following example reads the `/alive` endpoint anonymously. If the DS server's internal tests do not find errors that require administrative action, then it returns HTTP 200 OK:

```bash
$ curl --cacert ca-cert.pem --head https://localhost:8443/alive

HTTP/1.1 200 OK
...
```

If the server finds that it is subject to errors requiring administrative action, it returns HTTP 503 Service Unavailable.

If there are errors, anonymous users receive only the 503 error status. Error strings for diagnosis are returned as an array of `"alive-errors"` in the response body, but the response body is only returned to a user with the `monitor-read` privilege.

When a server returns `"alive-errors"`, diagnose and fix the problem, and then either restart or replace the server.

## Server health (HTTP)

The following example reads the `/healthy` endpoint anonymously. If the DS server is alive, as described in [Server is alive (HTTP)](#monitoring-liveness-http), any replication listener threads are functioning normally, and any replication delay is below the threshold configured as `max-replication-delay-health-check` (default: 5 seconds), then it returns HTTP 200 OK:

```bash
$ curl --cacert ca-cert.pem --head https://localhost:8443/healthy

HTTP/1.1 200 OK
...
```

If the server is subject to a replication delay above the threshold, then it returns HTTP 503 Service Unavailable. This result only indicates a problem if the replication delay is steadily high and increasing for the long term.

If there are errors, anonymous users receive only the 503 error status. Error strings for diagnosis are returned as an array of `"ready-errors"` in the response body, but the response body is only returned to a user with the `monitor-read` privilege.

When a server returns `"ready-errors"`, route traffic to another server until the current server is ready again.

## Server health (Prometheus)

In addition to the examples above, you can monitor whether a server is alive and able to handle requests as Prometheus metrics:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep health_status

# HELP ds_health_status_alive Indicates whether the server is alive
# TYPE ds_health_status_alive gauge
ds_health_status_alive 1.0
# HELP ds_health_status_healthy Indicates whether the server is able to handle requests
# TYPE ds_health_status_healthy gauge
ds_health_status_healthy 1.0
```

## Replication delay (Prometheus)

The following example reads a metric to check the delay in replication:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep receive_delay

# HELP ds_replication_replica_remote_replicas_receive_delay_seconds Current local delay in receiving replicated operations
# TYPE ds_replication_replica_remote_replicas_receive_delay_seconds gauge
ds_replication_replica_remote_replicas_receive_delay_seconds{<labels>} <delay>
```

DS replicas measure replication delay as the local delay when receiving and replaying changes. A replica calculates these local delays based on changes received from other replicas. Therefore, a replica can only calculate delays based on changes it has received. Network outages cause inaccuracy in delay metrics.

A replica calculates delay metrics based on times reflecting the following events:

* **t0**: the remote replica records the change in its data

* **t1**: the remote replica sends the change to a replica server

* **t2**: the local replica receives the change from a replica server

* **t3**: the local replica applies the change to its data

This figure illustrates when these events occur:

![Replication change processing events](../_images/repl-delay.svg)

Replication keeps track of changes using [change sequence numbers](../getting-started/glossary.html#csn) (CSNs), opaque and unique identifiers for each change that indicate when and where each change first occurred. The **tn** values are CSNs.

When the CSNs for the last change received and the last change replayed are identical, the replica has applied all the changes it has received. In this case, there is no known delay. The receive and replay delay metrics are set to 0 (zero).

When the last received and last replayed CSNs differ:

* Receive delay is set to the time **t2** - **t0** for the last change received.

  Another name for receive delay is current delay.

* Replay delay is approximately **t3** - **t2** for the last change replayed. In other words, it is an approximation of how long it took the last change to be replayed.

As long as replication delay tends toward zero regularly and over the long term, temporary spikes and increases in delay measurements are normal. When all replicas remain connected and yet replication delay remains high and increases over the long term, the high replication delay indicates a problem. Steadily high and increasing replication delay shows that replication is not converging, and the service is failing to achieve eventual consistency.

For a current snapshot of replication delays, you can also use the `dsrepl status` command. For details, see [replication status](../config-guide/replication.html#repl-status).

## Disk space (Prometheus)

The following example shows monitoring metrics you can use to check whether the server is running out of disk space:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep disk

# HELP ds_disk_free_space_bytes The amount of free disk space (in bytes)
# TYPE ds_disk_free_space_bytes gauge
ds_disk_free_space_bytes{disk="<partition>",} <bytes>
# HELP ds_disk_free_space_full_threshold_bytes The effective full disk space threshold (in bytes)
# TYPE ds_disk_free_space_full_threshold_bytes gauge
ds_disk_free_space_full_threshold_bytes{disk="<partition>",} <bytes>
# HELP ds_disk_free_space_low_threshold_bytes The effective low disk space threshold (in bytes)
# TYPE ds_disk_free_space_low_threshold_bytes gauge
ds_disk_free_space_low_threshold_bytes{disk="<partition>",} <bytes>
```

In your monitoring software, compare free space with the disk low and disk full thresholds. For database backends, these thresholds are set using the configuration properties: [disk-low-threshold](../configref/objects-je-backend.html#disk-low-threshold) and [disk-full-threshold](../configref/objects-je-backend.html#disk-full-threshold).

When you read from `cn=monitor` instead ,as described in [LDAP-based monitoring](ldap-monitoring.html), the relevant data are exposed on child entries of `cn=disk space monitor,cn=monitor`.

## Certificate expiration (Prometheus)

The following example shows how you can use monitoring metrics to check whether the server certificate is due to expire soon:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep cert

# HELP ds_certificates_certificate_expires_at_seconds Certificate expiration date and time
# TYPE ds_certificates_certificate_expires_at_seconds gauge
ds_certificates_certificate_expires_at_seconds{alias="ssl-key-pair",key_manager="PKCS12",} <sec_since_epoch>
```

In your monitoring software, compare the expiration date with the current date.

When you read from `cn=monitor` instead, as described in [LDAP-based monitoring](ldap-monitoring.html), the relevant data are exposed on child entries of `cn=certificates,cn=monitor`.

## Request statistics (Prometheus)

DS server connection handlers respond to client requests. The following example uses the default monitor user account to read statistics about client operations on each of the available connection handlers:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep connection_handlers
```

## Work queue (Prometheus)

DS servers have a work queue to track request processing by worker threads, and whether the server has rejected any requests due to a full queue. If enough worker threads are available, then no requests are rejected. The following example uses the default monitor user account to read statistics about the work queue:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep work_queue
```

To adjust the number of worker threads, see the settings for [Traditional Work Queue](../configref/objects-traditional-work-queue.html).

## Database size (Prometheus)

DS servers maintain counts of the number of entries in each backend. The following example uses the default monitor user account to read the counts:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep backend_entry_count
```

## Active users (Prometheus)

DS server connection handlers respond to client requests. The following example uses the default monitor user account to read active connections on each connection handler:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep "active_[cp]"
```

## Index use (Prometheus)

DS maintains metrics about index use. The metrics indicate how often an index was accessed since the DS server started.

The following example demonstrates how to read the metrics for all monitored indexes:

```bash
$ curl --cacert ca-cert.pem --user monitor:password https://localhost:8443/metrics/prometheus 2>/dev/null | grep index_uses
```

## Filtering results (Prometheus)

By default, DS servers return all Prometheus metrics. To limit what the server returns, set one of these HTTP endpoint properties for the `/metrics/prometheus`:

* [`excluded-metric-pattern`](../configref/objects-crest-metrics-endpoint.html#excluded-metric-pattern)

* [`included-metric-pattern`](../configref/objects-crest-metrics-endpoint.html#included-metric-pattern)

Set these properties to valid [Java regular expression patterns](https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/regex/Pattern.html).

The following configuration change causes the server to return only `ds_connection_handlers_ldap_requests_*` metrics. As mentioned in the reference documentation, "*The metric name prefix must not be included in the filter.*" Notice that the example uses `connection_handlers_ldap_requests`, not including the leading `ds_`:

```bash
$ dsconfig \
 set-http-endpoint-prop \
 --endpoint-name /metrics/prometheus \
 --set included-metric-pattern:'connection_handlers_ldap_requests' \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt
```

The following configuration change causes the server to exclude metrics whose names start with `ds_jvm_`. Notice that the example uses the regular expression `jvm_.*`:

```bash
$ dsconfig \
 set-http-endpoint-prop \
 --endpoint-name /metrics/prometheus \
 --set excluded-metric-pattern:'jvm_.*' \
 --hostname localhost \
 --port 4444 \
 --bindDN uid=admin \
 --bindPassword password \
 --usePkcs12TrustStore /path/to/opendj/config/keystore \
 --trustStorePassword:file /path/to/opendj/config/keystore.pin \
 --no-prompt
```