Monitor server health

Because IDM is highly modular and configurable, it is often difficult to assess whether a system has started up successfully, or whether the system is ready and stable after dynamic configuration changes have been made.

The health check service lets you monitor the status of internal resources through liveness and readiness probes. These probes can be useful for managing application lifecycle in containerized environments such as Kubernetes.

To monitor the status of external resources, such as LDAP servers and external databases, use the commands described in Check external system status over REST.

Liveness, readiness, and health probes

IDM includes endpoints for health checks, including a liveness probe, a readiness probe, and a general health probe. The openidm/health endpoint doesn’t generate audit logs.

liveness probe: Indicates whether the IDM instance is running. The probe is positive if the required bundles are installed and started. If the liveness probe fails, the instance might need to be restarted.
readiness probe: Indicates whether the instance is ready to accept traffic. The probe is positive if the required services are available. If the readiness probe fails, the instance shouldn’t receive traffic until the probe is positive.
health probe: A general check to determine the overall status of the IDM instance without generating audit logs.

Liveness probe

To check if the IDM instance is running, use the openidm/health/live endpoint. This endpoint returns a 200 OK status if the required bundles are installed and started. Otherwise, it returns a 503 Service Unavailable status.

IDM is alive
IDM is not alive

curl \
--user anonymous:anonymous \
"http://localhost:8080/openidm/health/live"
{}

curl \
--user anonymous:anonymous \
"http://localhost:8080/openidm/health/live"
{
  "code": 503,
  "reason": "Service Unavailable",
  "message": "Service Unavailable",
  "detail": {
    "shortDesc": "Inactive bundle 18 org.forgerock.openidm.external-email: RESOLVED; Found 1 inactive of 130 bundles for pattern org.forgerock.*",
    "state": "CRITICAL"
  }
}

For non-bearer token deployments, the openidm/health/live endpoint requires the basic authentication credential anonymous:anonymous. Bearer-token authenticated environments can call this endpoint without authentication.

Readiness probe

To check if the IDM instance is ready to accept traffic, use the openidm/health/ready endpoint. This endpoint returns a 200 OK status if the required services are available. Otherwise, it returns a 503 Service Unavailable status code when the health check readiness state is TEMPORARILY_UNAVAILABLE, CRITICAL, or HEALTHCHECK_UNKNOWN.

IDM is ready
IDM is TEMPORARILY_UNAVAILABLE
IDM is CRITICAL

curl \
--user anonymous:anonymous \
"http://localhost:8080/openidm/health/ready"
{}

curl \
--user anonymous:anonymous \
"http://localhost:8080/openidm/health/ready"
{
  "code": 503,
  "reason": "Service Unavailable",
  "message": "Service Unavailable",
  "detail": {
    "shortDesc": "IDM is temporarily unable to handle additional requests.",
    "state": "TEMPORARILY_UNAVAILABLE"
  }
}

curl \
--user anonymous:anonymous \
"http://localhost:8080/openidm/health/ready"
{
  "code": 503,
  "reason": "Service Unavailable",
  "message": "Service Unavailable",
  "detail": {
    "shortDesc": "IDM is in a critical state, unable to function properly.",
    "state": "CRITICAL"
  }
}

For non-bearer token deployments, the openidm/health/ready endpoint requires the basic authentication credential anonymous:anonymous. Bearer-token authenticated environments can call this endpoint without authentication.

The readiness probe includes a Jetty request queue health check that monitors the queue based on the webserver.json configuration property maxQueueSize:

If the queue is bounded (maxQueueSize > 0), the check returns TEMPORARILY_UNAVAILABLE when the queue reaches maximum capacity.
If the queue is unbounded (maxQueueSize <= 0), the check returns TEMPORARILY_UNAVAILABLE when the queue size reaches double the maxThreads value.

To prevent the state from rapidly switching back and forth, the status returns to OK only after the queue size drops below 50% of the triggering threshold.

Adjust liveness and readiness wait times

Use the configuration property openidm.healthcheck.executor.temporarilyAvailableGracePeriod to adjust how long the liveness and readiness probes waits before escalating a recurring TEMPORARILY_UNAVAILABLE state to CRITICAL. A TEMPORARILY_UNAVAILABLE status fails the readiness check and a CRITICAL status fails the liveness and readiness checks.

The default is 150000 milliseconds.

Health probe

This endpoint is deprecated. Use the liveness and readiness probes instead.

To verify the current server state, use the openidm/health endpoint:

curl \
--user anonymous:anonymous \
"http://localhost:8080/openidm/health"
{
  "_id": "",
  "_rev": "",
  "shortDesc": "OpenIDM ready",
  "state": "ACTIVE_READY"
}

The server can be in one of the following states:

`STARTING`	The server is starting up.
`ACTIVE_READY`	All specified requirements have been met to consider the server ready.
`ACTIVE_NOT_READY`	One or more specified requirements haven’t been met and the server isn’t considered ready.
`STOPPING`	The server is shutting down.

For non-bearer token deployments, the openidm/health endpoint requires the basic authentication credential anonymous:anonymous. Bearer-token authenticated environments can call this endpoint without authentication.

Legacy health probe

This endpoint is deprecated. Use the liveness and readiness probes instead.

The openidm/info/ping endpoint is the legacy health probe. It reports on the state of the server, outputs to the OSGi console, and generates audit log entries. To verify the current server state:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/info/ping"
{
  "_id": "",
  "_rev": "",
  "shortDesc": "OpenIDM ready",
  "state": "ACTIVE_READY"
}

Session information

To obtain information about the current IDM session, beyond basic health checks, use the following REST call:

curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
"http://localhost:8080/openidm/info/login"
{
  "_id": "login",
  "authenticationId": "openidm-admin",
  "authorization": {
    "userRolesProperty": "authzRoles",
    "component": "internal/user",
    "authLogin": false,
    "authenticationIdProperty": "username",
    "roles": [
      "internal/role/openidm-admin",
      "internal/role/openidm-authorized"
    ],
    "ipAddress": "0:0:0:0:0:0:0:1",
    "authenticationId": "openidm-admin",
    "id": "openidm-admin",
    "moduleId": "INTERNAL_USER",
    "queryId": "credential-internaluser-query"
  }
}

The precise output of this command depends on the authentication module responsible for authenticating the user. Learn more about Authentication and session modules.

Health check service

The configurable health check service verifies the status of the modules and services required for an operational system. During system startup, IDM checks that these modules and services are available and reports on any requirements that have not been met. If dynamic configuration changes are made, IDM rechecks that the required modules and services are functioning, to allow ongoing monitoring of system operation.

Examples of Required Modules

IDM checks all required modules. Examples of those modules are shown here:

"org.forgerock.openicf.framework.connector-framework"
"org.forgerock.openicf.framework.connector-framework-internal"
"org.forgerock.openicf.framework.connector-framework-osgi"
"org.forgerock.openidm.audit"
"org.forgerock.openidm.core"
"org.forgerock.openidm.enhanced-config"
"org.forgerock.openidm.external-email"
...
"org.forgerock.openidm.system"
"org.forgerock.openidm.ui"
"org.forgerock.openidm.util"
"org.forgerock.commons.org.forgerock.json.resource"
"org.forgerock.commons.org.forgerock.util"
"org.forgerock.openidm.security-jetty"
"org.forgerock.openidm.jetty-fragment"
"org.forgerock.openidm.quartz-fragment"
"org.ops4j.pax.web.pax-web-extender-whiteboard"
"org.forgerock.openidm.scheduler"
"org.ops4j.pax.web.pax-web-jetty"
"org.forgerock.openidm.repo-jdbc"
"org.forgerock.openidm.repo-ds"
"org.forgerock.openidm.config"
"org.forgerock.openidm.crypto"

Examples of Required Services

IDM checks all required services. Examples of those services are shown here:

"org.forgerock.openidm.config"
"org.forgerock.openidm.provisioner"
"org.forgerock.openidm.provisioner.openicf.connectorinfoprovider"
"org.forgerock.openidm.external.rest"
"org.forgerock.openidm.audit"
"org.forgerock.openidm.policy"
"org.forgerock.openidm.managed"
"org.forgerock.openidm.script"
"org.forgerock.openidm.crypto"
"org.forgerock.openidm.recon"
"org.forgerock.openidm.info"
"org.forgerock.openidm.router"
"org.forgerock.openidm.scheduler"
"org.forgerock.openidm.scope"
"org.forgerock.openidm.taskscanner"

You can replace the list of required modules and services, or add to it, by adding the following lines to your resolver/boot.properties file. Bundles and services are specified as a list of symbolic names, separated by commas:

openidm.healthservice.reqbundles - overrides the default required bundles.
openidm.healthservice.reqservices - overrides the default required services.
openidm.healthservice.additionalreqbundles - specifies required bundles (in addition to the default list).
openidm.healthservice.additionalreqservices - specifies required services (in addition to the default list).

By default, the server is given 15 seconds to start up all the required bundles and services before system readiness is assessed. This is not the total start time, but the time required to complete the service startup after the framework has started. You can change this default by setting the value of the servicestartmax property (in milliseconds) in your resolver/boot.properties file. This example sets the startup time to five seconds:

openidm.healthservice.servicestartmax=5000

PingIDM