Monitor services
The following sections describe how to set up and maintain monitoring in your deployment to ensure appropriate performance and service availability.
The Prometheus endpoint isn’t currently supported for Jakarta builds of the agent because of known issue AMAGENTS-6809. |
Monitor with Prometheus
Java Agent automatically exposes a monitoring endpoint where Prometheus can scrape metrics in a standard Prometheus format (version 0.0.4).
The Prometheus endpoint is enabled by default. You can disable it if required by setting the Enable Prometheus Monitoring property to false
.
You can find information about installing and running Prometheus in the Prometheus documentation.
Prometheus performance metrics are provided by an endpoint configured in the
protected web application’s web.xml
file. The endpoint must be accessible
to the Prometheus server that uses the performance data.
Tools such as Grafana are available to create customized charts and graphs based on the information collected by Prometheus. Learn more on the Grafana website. |
Expose the Prometheus endpoint
Use the following procedure to expose the Prometheus endpoint.
-
For each protected web application that is to expose metrics, edit the
web.xml
file for theagentapp
application.The following example exposes a base endpoint named
/metrics
:<servlet> <servlet-name>AgentMonitoring</servlet-name> <servlet-class>org.forgerock.http.servlet.HttpFrameworkServlet</servlet-class> <init-param> <param-name>application-loader</param-name> <param-value>service_loader</param-value> </init-param> </servlet> <servlet-mapping> <servlet-name>AgentMonitoring</servlet-name> <url-pattern>/metrics/*</url-pattern> </servlet-mapping>
Choose a name for the exposed base endpoint that does not conflict with any of the built-in agent endpoints.
-
Allow access to the Prometheus endpoint that is protected by the agent, in one of the following ways:
-
Configure Not-Enforced URIs to create a not-enforced URI rule for the base endpoint.
The following example rule allows access to the metrics base endpoint:
*/metrics/*
-
Configure Not-Enforced URIs, Not-Enforced Client IP List, and Not-Enforced Compound Rule Separator to create a compound not-enforced rule for the base endpoint.
The rule allows access from only the IP addresses of the Prometheus server.
The following example rule allows access to the
/metrics
endpoint for HTTP requests that come from the IP address range from 192.168.1.1 to 192.168.1.3:192.168.1.1-192.168.1.3 | */metrics/*
HTTP requests from other IP addresses are not able to access the metrics base endpoint.
-
Create an authorization policy in Identity Cloud or AM to restrict access to the metrics base endpoint.
The metric base endpoint doesn’t require login credentials. You can use a policy to ensure that requests to the endpoint are authenticated against the Identity Cloud or AM instance.
-
-
If the Prometheus endpoint is protected by Identity Cloud or AM policies, include the required credentials.
Access the Prometheus endpoint
When the example in Expose the Prometheus endpoint is configured, the
Prometheus endpoint is available at https://mydomain.example.com/agentapp/metrics/prometheus
.
-
Access the Prometheus endpoint as follows:
$ curl https://mydomain.example.com/agentapp/metrics/prometheus
The metrics are displayed:
# HELP ja_expired_session_cache_eviction the eviction count for the expired session cache # TYPE ja_expired_session_cache_eviction gauge ja_expired_session_cache_eviction 0.0 # HELP ja_expired_session_cache the hit count for the expired session cache # TYPE ja_expired_session_cache gauge ja_expired_session_cache{outcome="hit",} 0.0 ja_expired_session_cache{outcome="miss",} 0.0 # HELP ja_expired_session_cache_load the total load count for the expired session cache # TYPE ja_expired_session_cache_load gauge ...
Write metrics to CSV files
Configure Export Monitoring Metrics to CSV to write metric information to CSV files.
Monitoring types
This section describes the data types used in monitoring:
- Counter
-
Cumulative metric for a numerical value that only increases.
- Distinct counter
-
Metric that provides an estimate of the number of unique values recorded.
For example, use this to estimate the number of unique users who have authenticated, or unique client IP addresses.
The
DistinctCounter
type is reported as aGauge
type.The
DistinctCounter
metric is calculated per instance of AM and can’t be aggregated across multiple instances to get a site-wide view. - Gauge
-
Metric for a numerical value that can increase or decrease.
The value for a gauge is calculated when requested and represents the state of the metric at that specific time.
- Timer
-
Metric that provides duration information.
The Prometheus endpoint doesn’t provide rate-based statistics because rates can be calculated from the time-series data. Instead, the Prometheus endpoint includes summary metrics whose names have the following suffixes or labels:
-
count
: number of events recorded -
sum
: sum of the durations recorded -
{quantile="0.5"}
: 50% of the durations are at or below this value -
{quantile="0.75"}
: 75% of the durations are at or below this value -
{quantile="0.95"}
: 95% of the durations are at or below this value -
{quantile="0.98"}
: 98% of the durations are at or below this value -
{quantile="0.99"}
: 99% of the durations are at or below this value -
{quantile="0.999"}
: 99.9% of the durations are at or below this value
The
Timer
metric type is reported as aSummary
type.Duration-based quantile values are weighted towards newer data. By representing approximately the last five minutes of data, the timers make it easier to see recent changes in behavior, rather than a uniform average of recordings since the server was started.
-
Metrics at the Prometheus endpoint
Audit handler metrics
Java Agent exposes the following audit handler monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Timer |
Time taken to generate an audit object. |
|
Timer |
Time taken to audit outcomes, both locally to the agent and remotely in Identity Cloud or AM. |
Label | Values |
---|---|
|
|
|
|
Endpoint and REST SDK metrics
Java Agent exposes the following endpoint and REST SDK monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Timer |
Time taken to retrieve user session information from Identity Cloud or AM. |
|
Timer |
Time taken to retrieve the user profile information from Identity Cloud or AM. |
|
Timer |
Time taken to retrieve policy decisions from Identity Cloud or AM. |
Expired session metrics
Java Agent exposes the following expired session monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Total number of entries in the expired session cache. |
|
Gauge |
Eviction count for the expired session cache. |
|
Gauge |
Load count for the expired session cache. |
|
Gauge |
Load time for the expired session cache, in milliseconds. |
|
Gauge |
Hit count for the expired session cache. |
|
Gauge |
Miss count for the expired session cache. |
JSON Web Token (JWT) metrics
Java Agent exposes the following JWT-related monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Size of the JWT cache. |
|
Gauge |
The eviction count for the JWT cache. |
|
Gauge |
The load count for the JWT cache. |
|
Gauge |
The load time for the JWT cache, in milliseconds. |
|
Gauge |
The hit count for the JWT cache. |
|
Gauge |
The miss count for the JWT cache. |
JVM metrics
Java Agent exposes the following JVM-related monitoring metrics:
Some |
Metric | Type | Description |
---|---|---|
|
Gauge |
Number of processors available to the Java virtual machine. |
|
Gauge |
Number of classes loaded since the Java virtual machine started. |
|
Gauge |
Number of classes unloaded since the Java virtual machine started. |
|
Gauge |
Bytes of free memory allocated to the Java virtual machine |
|
Gauge |
Number of collections performed by the "G1 old generation" garbage collection algorithm. |
|
Gauge |
Approximate accumulated time taken by the "G1 old generation" garbage collection algorithm. |
|
Gauge |
Number of collections performed by the "G1 young generation" garbage collection algorithm. |
|
Gauge |
Approximate accumulated time taken by the "G1 young generation" garbage collection algorithm. |
|
Gauge |
Maximum amount of memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of heap memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of heap memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of heap memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of heap memory used by the Java virtual machine. |
|
Gauge |
Amount of heap memory allocated to the Java virtual machine. |
|
Gauge |
Amount of memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of non-heap memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of non-heap memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of non-heap memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of non-heap memory used by the Java virtual machine. |
|
Gauge |
Amount of non-heap memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "non-method code cache heap" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "non-method code cache heap" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "non-method code cache heap" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "non-method code cache heap" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "non-method code cache heap" memory used by the Java virtual machine. |
|
Gauge |
Amount of "non-profiled code cache heap" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "non-profiled code cache heap" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "non-profiled code cache heap" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "non-profiled code cache heap" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "non-profiled code cache heap" memory used by the Java virtual machine. |
|
Gauge |
Amount of "profiled code cache heap" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "profiled code cache heap" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "profiled code cache heap" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "profiled code cache heap" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "profiled code cache heap" memory used by the Java virtual machine. |
|
Gauge |
Amount of "compressed class space" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "compressed class space" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "compressed class space" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "compressed class space" memory used by the Java virtual machine. |
|
Gauge |
Amount of "compressed class space" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "G1 eden space" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "G1 eden space" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "G1 eden space" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "G1 eden space" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "G1 eden space" memory used by the Java virtual machine. |
|
Gauge |
Amount of "G1 eden space" memory used by the Java virtual machine after garbage collection. |
|
Gauge |
Amount of "G1 old generation" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "G1 old generation" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "G1 old generation" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "G1 old generation" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "G1 old generation" memory used by the Java virtual machine. |
|
Gauge |
Amount of "G1 old generation" memory used by the Java virtual machine after garbage collection. |
|
Gauge |
Amount of "G1 survivor space" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "G1 survivor space" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "G1 survivor space" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "G1 survivor space" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of "G1 survivor space" memory used by the Java virtual machine. |
|
Gauge |
Amount of "G1 survivor space" memory used by the Java virtual machine after garbage collection. |
|
Gauge |
Amount of "metaspace" memory the Java virtual machine initially requested from the operating system. |
|
Gauge |
Maximum amount of "metaspace" memory the Java virtual machine attempts to use. |
|
Gauge |
Amount of "metaspace" memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of "metaspace" memory used by the Java virtual machine. |
|
Gauge |
Amount of "metaspace" memory allocated to the Java virtual machine. |
|
Gauge |
Amount of memory committed for the Java virtual machine to use. |
|
Gauge |
Amount of memory used by the Java virtual machine. |
|
Gauge |
Number of threads in the BLOCKED state. |
|
Gauge |
Number of live threads including both daemon and non-daemon threads. |
|
Gauge |
Number of live daemon threads. |
|
Gauge |
Number of threads in the NEW state. |
|
Gauge |
Number of threads in the RUNNABLE state. |
|
Gauge |
Number of threads in the TERMINATED state. |
|
Gauge |
Number of threads in the TIMED_WAITING state. |
|
Gauge |
Number of threads in the WAITING state. |
|
Gauge |
Amount of memory used by the Java virtual machine |
Not-enforced rule metrics
Java Agent exposes the following not-enforced rule monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Size of the not-enforced URI matched cache. |
|
Gauge |
Eviction count for the not-enforced URI matched cache. |
|
Gauge |
Load count for the not-enforced URI matched cache. |
|
Gauge |
Load time for the not-enforced URI matched cache, in milliseconds. |
|
Gauge |
Hit count for the not-enforced URI matched cache. |
|
Gauge |
Miss count for the not-enforced URI matched cache. |
|
Gauge |
Size of the not-enforced URI unmatched cache. |
|
Gauge |
Eviction count for the not-enforced URI unmatched cache. |
|
Gauge |
Load count for the not-enforced URI unmatched cache. |
|
Gauge |
Load time for the not-enforced URI unmatched cache, in milliseconds. |
|
Gauge |
Hit count for the not-enforced URI unmatched cache. |
|
Gauge |
Miss count for the not-enforced URI unmatched cache. |
|
Gauge |
Size of the not-enforced IP matched cache. |
|
Gauge |
Eviction count for the not-enforced IP matched cache. |
|
Gauge |
Load count for the not-enforced IP matched cache. |
|
Gauge |
Load time for the not-enforced IP matched cache, in milliseconds. |
|
Gauge |
Hit count for the not-enforced IP matched cache. |
|
Gauge |
Miss count for the not-enforced IP matched cache. |
|
Gauge |
Size of the not-enforced IP unmatched cache. |
|
Gauge |
Eviction count for the not-enforced IP unmatched cache. |
|
Gauge |
Load count for the not-enforced IP unmatched cache. |
|
Gauge |
Load time for the not-enforced IP unmatched cache, in milliseconds. |
|
Gauge |
Hit count for the not-enforced IP unmatched cache. |
|
Gauge |
Miss count for the not-enforced IP unmatched cache. |
|
Gauge |
Size of the not-enforced compound matched cache. |
|
Gauge |
Eviction count for the not-enforced compound matched cache. |
|
Gauge |
Load count for the not-enforced compound matched cache. |
|
Gauge |
Load time for the not-enforced compound matched cache, in milliseconds. |
|
Gauge |
Hit count for the not-enforced compound matched cache. |
|
Gauge |
Miss count for the not-enforced compound matched cache. |
|
Gauge |
Size of the not-enforced compound unmatched cache. |
|
Gauge |
Eviction count for the not-enforced compound unmatched cache. |
|
Gauge |
Load count for the not-enforced compound unmatched cache. |
|
Gauge |
Load time for the not-enforced compound unmatched cache, in milliseconds. |
|
Gauge |
Hit count for the not-enforced compound unmatched cache. |
|
Gauge |
Miss count for the not-enforced compound unmatched cache. |
Policy decision metrics
Java Agent exposes the following policy decision monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Size of the policy decision cache. |
|
Gauge |
Eviction count for the policy decision cache. |
|
Gauge |
Load count for the policy decision cache. |
|
Gauge |
Load time for the policy decision cache, in milliseconds. |
|
Gauge |
Hit count for the policy decision cache. |
|
Gauge |
Miss count for the policy decision cache. |
POST data preservation metrics
Java Agent exposes the following POST data preservation monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Size of the POST data preservation cache. |
|
Gauge |
Eviction count for the POST data preservation cache. |
|
Gauge |
Load count for the POST data preservation cache. |
|
Gauge |
Load time for the POST data preservation cache, in milliseconds. |
|
Gauge |
Hit count for the POST data preservation cache. |
|
Gauge |
Miss count for the POST data preservation cache. |
Request metrics
Java Agent exposes the following request monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Timer |
Rate of granted/denied requests and their decision. |
Label | Values |
---|---|
|
|
|
|
Session information metrics
Java Agent exposes the following session information monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Size of the session information cache. |
|
Gauge |
Eviction count for the session information cache. |
|
Gauge |
Load count for the session information cache. |
|
Gauge |
Load time for the session information cache, in milliseconds. |
|
Gauge |
Hit count for the session information cache. |
|
Gauge |
Miss count for the session information cache. |
SSO token to JWT exchange metrics
Java Agent exposes the following SSO token to JWT exchange monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Size of the SSO token exchange cache. |
|
Gauge |
Eviction count for the SSO token exchange cache. |
|
Gauge |
Load count for the SSO token exchange cache. |
|
Gauge |
Load time for the SSO token exchange, in milliseconds. |
|
Gauge |
Hit count for the SSO token exchange cache. |
|
Gauge |
Miss count for the SSO token exchange cache. |
WebSocket metrics
Java Agent exposes the following WebSocket-related monitoring metrics:
Metric | Type | Description |
---|---|---|
|
Gauge |
Number of milliseconds since anything was received over the WebSocket, for example, a ping or a notification. |
|
Gauge |
Number of milliseconds since anything was sent over the WebSocket. |
|
Counter |
Count of WebSocket configuration change notifications received.(1) |
|
Count of WebSocket configuration change notifications processed. |
|
|
Counter |
Count of WebSocket policy change notifications received.(1) |
|
Counter |
Count of WebSocket policy change notifications processed. |
|
Counter |
Count of WebSocket session logout notifications received.(1) |
|
Counter |
Count of WebSocket session logout notifications processed. |
|
Timer |
Ping/pong round trip time. |
(1) Some can be ignored if the realm or agent name isn’t applicable.