Monitoring and metrics
Configure ForgeRock® Identity Management server logs and monitoring metrics.
ForgeRock Identity Platform™ serves as the basis for our simple and comprehensive Identity and Access Management solution. We help our customers deepen their relationships with their customers, and improve the productivity and connectivity of their employees and partners. For more information about ForgeRock and about the platform, see https://www.forgerock.com.
The ForgeRock Common REST API works across the platform to provide common ways to access web resources and collections of resources.
Server logs
Server logging is not the same as auditing. Auditing logs activity on the IDM system, such as access, and synchronization. Server logging records information about the internal workings of IDM, like system messages, error reporting, service loading, or startup and shutdown messaging.
Configure server logging in your project’s conf/logging.properties
file. Changes to logging settings require a server restart before they take effect. Alternatively, use JMX via jconsole to change the logging settings. In this case, changes take effect without restarting the server.
Log message handlers
The way IDM logs messages is set in the handlers
property in the logging.properties
file. This property has the following value by default:
handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler
The default handlers are:
-
FileHandler
writes formatted log records to a single file or to a set of rotating log files. By default, log files are written tologs/openidm*.log
files. -
ConsoleHandler
writes formatted logs toSystem.err
.
Additional log message handlers are listed in the logging.properties
file.
Log message format
IDM supports the two default log formatters included with Java. These are set in the conf/logging.properties
file:
-
java.util.logging.SimpleFormatter.format
outputs a text log file that is human-readable. This is the default formatter. -
java.util.logging.XMLFormatter
outputs logs as XML, for use in logging software that can read XML logs.
IDM extends the Java SimpleFormatter
with the following formatting options:
org.forgerock.openidm.logger.SanitizedThreadIdLogFormatter
-
This is the default formatter for console and file logging. It extends the
SimpleFormatter
to include the thread ID of the thread that generated each message. The thread ID helps with debugging when reviewing the logs.In the following example log excerpt, the thread ID is
[19]
:[19] May 23, 2018 10:30:26.959 AM org.forgerock.openidm.repo.opendj.impl.Activator start INFO: Registered bootstrap repository service [19] May 23, 2018 10:30:26.960 AM org.forgerock.openidm.repo.opendj.impl.Activator start INFO: DS bundle started
The SanitizedThreadIdLogFormatter
also encodes all control characters (such as newline characters) using URL-encoding, to protect against log forgery. Control characters in stack traces are not encoded. org.forgerock.openidm.logger.ThreadIdLogFormatter
-
Similar to the
SanitizedThreadIdLogFormatter
, but does not encode control characters. If you do not want to encode control characters in file and console log messages, edit the file and console handlers inconf/logging.properties
as follows:java.util.logging.FileHandler.formatter = org.forgerock.openidm.logger.ThreadIdLogFormatter java.util.logging.ConsoleHandler.formatter = org.forgerock.openidm.logger.ThreadIdLogFormatter
properties
The SimpleFormatter
(and, by extension, the SanitizedThreadIdLogFormatter
and ThreadIdLogFormatter
) lets you customize what information to include in log messages, and how this information is laid out. By default, log messages include the date, time (down to the millisecond), log level, source of the message, and the message sent (including exceptions). To change the defaults, adjust the value of java.util.logging.SimpleFormatter.format
in your conf/logging.properties
file. For more information on how to customize the log message format, see the related Java documentation.
Logging level
By default, IDM logs messages at the INFO
level. This logging level is specified with the following global property in conf/logging.properties
:
.level=INFO
You can specify different separate logging levels for individual server features which override the global logging level. Set the log level, per package to one of the following:
SEVERE (highest value)
WARNING
INFO
CONFIG
FINE
FINER
FINEST (lowest value)
For example, the following setting decreases the messages logged by the embedded PostgreSQL database:
# reduce the logging of embedded postgres since it is very verbose
ru.yandex.qatools.embed.postgresql.level = SEVERE
Set the log level to OFF
to disable logging completely (Disable Logs), or to ALL
to capture all possible log messages.
If you use logger
functions in your JavaScript scripts, set the log level for the scripts as follows:
org.forgerock.openidm.script.javascript.JavaScript.level=level
You can override the log level settings, per script, with the following setting:
org.forgerock.openidm.script.javascript.JavaScript.script-name.level=level
For more information about using logger
functions in scripts, see Log Functions.
It is strongly recommended that you do not log messages at the |
Log file rotation
By default, IDM rotates log files when the size reaches 5 MB, and retains up to 5 files. All system and custom log messages are also written to these files. You can modify these limits in the following properties in the logging.properties
file for your project:
# Limiting size of output file in bytes:
java.util.logging.FileHandler.limit = 5242880
# Number of output files to cycle through, by appending an
# integer to the base file name:
java.util.logging.FileHandler.count = 5
There is currently no |
Disable logs
If necessary, you can disable logs. For example, to disable ConsoleHandler
logging, make the following changes in your project’s conf/logging.properties
file before you start IDM.
Set java.util.logging.ConsoleHandler.level = OFF
, and comment out other references to ConsoleHandler
, as shown in the following excerpt:
# ConsoleHandler: A simple handler for writing formatted records to System.err
#handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler
handlers=java.util.logging.FileHandler
...
# --- ConsoleHandler ---
# Default: java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.level = OFF
#java.util.logging.ConsoleHandler.formatter = ...
#java.util.logging.ConsoleHandler.filter=...
Monitoring
IDM includes the following tools for monitoring metrics:
-
A Dropwizard dashboard widget, for viewing metrics within IDM.
-
A Prometheus endpoint, for viewing metrics through external resources such as Prometheus and Grafana.
The tool that you choose will depend on the metrics you monitor and on what you plan to use the metrics for.
IDM does not gather metrics by default. To enable metrics gathering, open conf/metrics.json
and set the enabled
property to true
.
To verify that metrics are successfully enabled, run:
curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request GET \ 'http://localhost:8080/openidm/metrics/api?_queryFilter=true'
If you have enabled metrics, this command returns information about the metrics that are being collected. Otherwise, the command returns a bad request response.
Metrics are reported only after they have been triggered by activity in IDM, such as a reconciliation.
For a list of available metrics, see Metrics reference.
Dropwizard widget
The Dropwizard widget creates a graph of metrics based on server activity.
To add the Dropwizard widget, click Add Widget on any Dashboard in the admin UI, and select Dropwizard Table with Graph from the list.
The Dropwizard widget is useful for lightweight, live monitoring of IDM; however, it has a few limitations:
-
The graphs created by the widget do not persist; they are restarted if you reload or navigate away from the page.
-
The widget only works with time-based metrics (other forms of metrics are not displayed in the widget).
Prometheus endpoint
Prometheus is a third-party tool used for gathering and processing monitoring data. Prometheus uses the openidm/metrics/prometheus
endpoint to gather information. This endpoint is protected by a basic authentication filter, using the following credentials, set in the resolver/boot.properties
file:
openidm.prometheus.username=username
openidm.prometheus.password=password
This section describes how to install Prometheus and Grafana to collect IDM metrics. These third-party tools are not supported by ForgeRock. For more information running Prometheus, see the Prometheus documentation.
Configure Prometheus
You must configure Prometheus using a prometheus.yml
configuration file. For more information, see the Prometheus configuration documentation. An example prometheus.yml
file would be:
global:
scrape_interval: 15s
external_labels:
monitor: 'my_prometheus'
# https://prometheus.io/docs/operating/configuration/#scrape_config
scrape_configs:
- job_name: 'openidm'
scrape_interval: 15s
scrape_timeout: 5s
metrics_path: 'openidm/metrics/prometheus'
scheme: http
basic_auth:
username: 'prometheus'
password: 'prometheus'
static_configs:
- targets: ['localhost:8080']
This example configures Prometheus to poll the IDM endpoint every 5 seconds (scrape_interval: 5s
), receiving metrics in a plain text format (_fields: ['text']
and _mimeType: ['text/plain;version=0.0.4']
). For more information about reporting formats, see the Prometheus documentation on Exposition Formats. Use curl
to validate that this configuration returns the expected results:
curl \ --user prometheus:prometheus \ --header "Accept-API-Version: resource=1.0" \ --request GET \ 'http://localhost:8080/openidm/metrics/prometheus'
Start Prometheus with your prometheus.yml
configuration file by running:
prometheus --config.file=/path/to/prometheus.yml
You can confirm that Prometheus is gathering data from IDM by navigating to the Prometheus monitoring page (by default, http://localhost:9090
).
Configure Grafana
Prometheus lets you monitor and process information provided by IDM. If you need deeper analytics, you can use tools such as Grafana to create customized charts and graphs based on Prometheus data. For information on installing and running Grafana, see the Grafana website.
The default username and password for Grafana are both |
To set up a Grafana dashboard with IDM metrics using Prometheus, add your Prometheus installation to Grafana, as a data source. Click Configuration > Data Sources from the left navigation panel in Grafana, and click Add Data Source.
Complete the options on the Add Data Source screen. The options in this list match the settings in the monitoring.dashboard.json
file.
-
Give your data source a name, in this case,
ForgeRockIDM
. -
Set the Type to Prometheus.
-
Set the URL (by default,
http://localhost:9090
). -
Set Access to Server (default).
-
Enable Basic Auth.
-
Set a username and password of
prometheus
. -
Click Save, and click Test Connection. If the configuration succeeds, you’ll see the following message: "Data source is working."
When Prometheus has been set up as a data source in Grafana, you can create a dashboard with IDM metrics. Create a dashboard in one of the following ways:
-
Download the Monitoring Dashboard Samples from the ForgeRock BackStage download site. Find
monitoring.dashboard.json
in the downloaded .zip file. In the Grafana administrative screen, click Home > Import, and import the noted file. -
Click Create > Dashboard.
-
Click Graph.
-
Click Panel Title > Edit.
-
Enter the metrics to display (which will be available in autocomplete as you type), or build more complex queries using the Prometheus query language.
-
Metrics reference
IDM exposes a number of metrics. All metrics are available at both the openidm/metrics/api
and openidm/metrics/prometheus
endpoints. The actual metric names can vary, depending on the endpoint used. Also see Monitoring.
Metric types
Metrics are organized into the following types:
Timer
Timers provide a histogram of the duration of an event, along with a measure of the rate of occurrences. Timers can be monitored using the Dropwizard dashboard widget and the IDM Prometheus endpoint. Durations in timers are measured in milliseconds. Rates are reported in number of calls per second. The following example shows a Timer metric:
{
"_id": "sync.source.perform-action",
"count": 2,
"max": 371.53391,
"mean": 370.1752705,
"min": 368.816631,
"p50": 371.53391,
"p75": 371.53391,
"p95": 371.53391,
"p98": 371.53391,
"p99": 371.53391,
"p999": 371.53391,
"stddev": 1.3586395,
"m15_rate": 0.393388581528647,
"m1_rate": 0.311520313228562,
"m5_rate": 0.3804917698002856,
"mean_rate": 0.08572717156016606,
"duration_units": "milliseconds",
"rate_units": "calls/second",
"total": 740.350541,
"_type": "timer"
}
Summary
Summaries are similar to Timers in that they measure a distribution of events. However, Summaries record values that aren’t units of time, such as user login counts. Summaries cannot be graphed in the Dropwizard dashboard widget, but are available through the Prometheus endpoint, and by querying the openidm/metrics/api
endpoint directly. The following example shows a Summary metric:
{
"_id": "audit.recon",
"m15_rate": 0.786777163057294,
"m1_rate": 0.623040626457124,
"m5_rate": 0.7609835396005712,
"mean_rate": 0.16977218861919927,
"units": "events/second",
"total": 4,
"count": 4,
"_type": "summary"
}
Gauge
Gauge metrics return a numerical value that can increase or decrease. The value for a gauge is calculated on request, and represents the state of the metric at that specific time. The following example shows a Gauge metric:
{
"_id": "jvm.used-memory",
"value": 2147483648,
"_type": "gauge"
}
API metrics
Metrics accessed at the api
endpoint (such as those consumed by the Dropwizard dashboard widget) use dot notation for their metric names; for example, recon.target-phase
. The following table lists the API metrics available in IDM:
API metrics available in IDM
API Metric Name | Type | Description |
---|---|---|
|
Summary |
Count of all audit events generated of a given topic type. |
|
Timer |
Rate of reading response objects, to fulfill the |
|
Timer |
Rate of reading response objects, to fulfill the |
|
Timer |
Rate at which filter scripts are executed, per action. Monitors scripted filters and delegated admin. |
|
Timer |
Rate of ICF query executions with queryExpression, and time taken to perform this operation. |
|
Timer |
Rate of ICF query executions with queryFilter, and time taken to perform this operation. |
|
Timer |
Rate of ICF query executions with queryId, and time taken to perform this operation. |
|
Timer |
Rate of ICF query executions when the query type is UNKNOWN, and time taken to perform this operation. |
|
Timer |
Rate of operations on internal objects. |
|
Timer |
Rate of fetch operations of relationship fields for internal objects. |
|
Timer |
Query rate on relationship values for internal objects. |
|
Timer |
Rate of script executions on internal object. |
|
Timer |
Rate of validate operations of relationship fields for internal objects. |
|
Timer |
Rate of responses requiring field augmentation. When the repository cannot retrieve all data in a single call, IDM performs additional read operations to complete (augment) the missing data. |
|
Timer |
Rate of operations on a managed object. |
|
Timer |
Rate of fetches of relationship fields of a managed object. |
|
Timer |
Rate of queries to get relationship values for a resource on a managed object. |
|
Timer |
Rate of validations of relationship fields of a managed object. |
|
Timer |
Rate of executions of a script on a managed object. |
|
Timer |
Latency of enforcing temporal constraints on role objects during object creation. |
|
Timer |
Latency of enforcing temporal constraints on role objects during object deletion. |
|
Timer |
Latency of enforcing temporal constraints on role objects during object update. |
|
Timer |
Latency of enforcing temporal constraints on relationship grants during edge creation. |
|
Timer |
Latency of enforcing temporal constraints on relationship grants during edge deletion. |
|
Timer |
Latency of enforcing temporal constraints on relationship grants during edge update. |
|
Timer |
Rate of reads on relationship endpoint edges for validation. |
|
Timer |
Time spent in filter which maps non-nullable, null-valued array fields to an empty array. This filter is traversed for all repo access relating to internal and managed objects. |
|
Timer |
Rate of executions of a full reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of merge operations after source and/or target objects have been retrieved during a merged query of recon association entries. |
|
Timer |
Rate of individual paged recon association entry queries during a merged query. More than one page of entries might be requested to build a single page of merged results. |
|
Timer |
Rate of source object retrieval via query when merging source objects to recon association entries. |
|
Timer |
Rate of target object retrieval via query when merging target objects to recon association entries. |
|
Timer |
The time taken to persist association data. The operation can be |
|
Timer |
Rate of executions of the id query phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of executions of the source phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of pagination executions of the source phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of executions of the target phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Time (ms) spent running the Edge→Vertex relationship join query on the database and collecting the result set. |
|
Timer |
Rate of relationship graph query execution times. |
|
Timer |
Rate of relationship graph query result processing times. |
|
Timer |
Rate of executions of a query with queryId at a repository level, and time taken to perform this operation. |
|
Count |
Counts the usage statistics of the |
|
Timer |
Rate of retrievals of a repository connection. |
|
Timer |
Rate of actions to a repository datasource for a generic/explicit mapped table. |
|
Timer |
Rate of filtered queries (using native query expressions) on the relationship table. This metric measures the time spent making the query (in ms), and the number of times the query is invoked. |
|
Timer |
Rate of filtered queries (using the |
|
Timer |
Rate of execution time on the JDBC database for the |
|
Timer |
Rate of execution time on the JDBC database for CRUD operations. This rate does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool. |
|
Timer |
Rate of execution time on the JDBC database for queries (either |
|
Timer |
Rate of CRUDPAQ operations to a repository datasource for a generic/explicit/relationship mapped table. |
|
Timer |
Time (ms) spent in the various phases to retrieve relationship expanded data referenced by queried objects. |
|
Timer |
Rate of initiations of a CRUDPAQ operation to a repository datasource. |
|
Timer |
Rate of actions over the router, and time taken to perform this operation. |
|
Timer |
Rate of creates over the router, and time taken to perform this operation. |
|
Timer |
Rate of deletes over the router, and time taken to perform this operation. |
|
Timer |
Rate of patches over the router, and time taken to perform this operation. |
|
Timer |
Rate of queries with queryExpression completed over the router, and time taken to perform this operation. |
|
Timer |
Rate of queries with queryFilter completed over the router, and time taken to perform this operation. |
|
Timer |
Rate of reads over the router, and time taken to perform this operation. |
|
Timer |
Rate of updates over the router, and time taken to perform this operation. |
|
Timer |
Time spent storing scheduled jobs in the repository. |
|
Timer |
Execution rate of scheduler requests. |
|
Timer |
Rate of calls to a script and time taken to complete. |
|
Summary |
Count of all successful user self-service password resets. |
|
Summary |
Count of all successful user self-service registrations by registration type. |
|
Summary |
Count of all successful user self-service registrations by registration type and provider. |
|
Timer |
Rate of requests to create a target object, and time taken to perform the operation. |
|
Timer |
Rate of requests to delete a target object, and time taken to perform the operation. |
|
Timer |
Rate of configurations applied to a mapping. |
|
Timer |
Rate of acquisition of queued synchronization events from the queue. |
|
Timer |
Rate of deletion of synchronization events from the queue. |
|
Timer |
Rate at which queued synchronization operations are executed. |
|
Summary |
Number of queued synchronization operations that failed. |
|
Summary |
Number of queued synchronization events that were acquired by another node in the cluster. |
|
Summary |
Number of queued synchronization events that were rejected because the backing thread-pool queue was at full capacity and the thread-pool had already allocated its maximum configured number of threads. |
|
Timer |
Rate at which queued synchronization events are released. |
|
Timer |
Times the release of queued synchronization events after a failure and before exceeding the retry count. |
|
Timer |
Rate of insertion of synchronization events into the queue. |
|
Timer |
The latency involved in polling for synchronization events. |
|
Timer |
Rate of reads of an object. |
|
Timer |
Rate of assessments of a synchronization situation. |
|
Timer |
Rate of correlations between a target and a given source, and time taken to perform this operation. |
|
Timer |
Rate of determinations done on a synchronization action based on its current situation. |
|
Timer |
Rate of completions of an action performed on a synchronization operation. |
|
Timer |
Rate of assessments of a target situation. |
|
Timer |
Rate of determinations done on a target action based on its current situation. |
|
Timer |
Rate of completions of an action performed on a target sync operation. |
|
Timer |
Rate of requests to update an object on the target, and the time taken to perform this operation. |
|
Summary |
Count of all successful logins by user type. |
|
Summary |
Count of all successful logins by user type and provider. |
|
Summary |
Number of 404 responses encountered when querying the |
|
Summary |
Number of edges skipped due to an unsatisfied temporal constraint on either the edge or the referred-to vertex. Encountered when querying the resource collection and relationship field at the traversal_depthX tag for the most recent X. |
|
Timer |
Time spent traversing relationship fields to calculate the specified virtual properties. The managed objects linked to by the traversal relationship fields define a tree, whose root is the virtual property host. This object tree is traversed depth-first, with the traversal_depthX corresponding to the latency involved with each relationship traversal. Traversal_depth0 corresponds to the first relationship field traversed. Because the tree is traversed depth-first, traversal_depthX will subsume all the traversal latencies for all traversal_depth Y, where Y>X. |
API JVM metrics available in IDM
These metrics depend on the JVM version and configuration. In particular, garbage-collector-related metrics depend on the garbage collector that the server uses. The garbage-collector metric names are unstable, and can change even in a minor JVM release. |
API Metric Name | Type | Unit | Description |
---|---|---|---|
|
Gauge |
Count |
Number of processors available to the JVM. For more information, see Runtime. |
|
Gauge |
Count |
Number of classes loaded since the Java virtual machine started. For more information, see ClassLoadingMXBean. |
|
Gauge |
Count |
Number of classes unloaded since the Java virtual machine started. For more information, see ClassLoadingMXBean. |
|
Gauge |
Bytes |
For more information, see Runtime. |
|
Gauge |
Count |
For each garbage collector in the JVM. For more information, see GarbageCollectorMXBean. |
|
Gauge |
Milliseconds |
|
|
Gauge |
Count |
|
|
Gauge |
Milliseconds |
|
|
Gauge |
Bytes |
For more information, see Runtime. |
|
Gauge |
Bytes |
Amount of heap memory committed for the JVM to use. For more information, see MemoryMXBean. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Maximum amount of heap memory available to the JVM. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Amount of heap memory used by the JVM. |
|
Gauge |
Bytes |
Amount of non-heap memory committed for the JVM to use. |
|
Gauge |
Bytes |
Amount of non-heap memory the JVM initially requested from the operating system. |
|
Gauge |
Bytes |
Maximum amount of non-heap memory available to the JVM. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Amount of non-heap memory used by the JVM. |
|
Gauge |
Bytes |
For each pool. For more information, see MemoryPoolMXBean. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Amount of memory that is committed for the JVM to use. For more information, see MemoryMXBean. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Count |
For more information, see ThreadMXBean. |
|
Gauge |
Count |
Number of live threads including both daemon and non-daemon threads. |
|
Gauge |
Count |
Number of live daemon threads. |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Bytes |
For more information, see totalMemory(). |
API workflow metrics available in IDM
API Metric Name | Type | Description |
---|---|---|
|
Timer |
Time spent invoking a message event. |
|
Timer |
Time spent invoking a signal event. |
|
Timer |
Time spent triggering an execution. |
|
Timer |
Time spent querying executions. |
|
Timer |
Time spent forcing synchronous execution of a job. |
|
Timer |
Time spent displaying the stacktrace for a job that triggered an exception. |
|
Timer |
Time spent deleting a job. |
|
Timer |
Time spent querying jobs. |
|
Timer |
Time spent reading a single job. |
|
Timer |
Time spent to execute dead-letter job. |
|
Timer |
Time spent to retrieve the stacktrace for a dead-letter job. |
|
Timer |
Time spent to delete a dead letter job. |
|
Timer |
Time spent to query dead letter jobs. |
|
Timer |
Time spent to read a dead letter job. |
|
Timer |
Time spent to deploy a model. |
|
Timer |
Time spent to list model deployments. |
|
Timer |
Time spent to validate BPMN content. |
|
Timer |
Time spent to create a model. |
|
Timer |
Time spent to delete a model. |
|
Timer |
Time spent to query models. |
|
Timer |
Time spent to read a model. |
|
Timer |
Time spent to update a model. |
|
Timer |
Time spent to delete a process definition. |
|
Timer |
Time spent to query process definitions. |
|
Timer |
Time spent to read a process definition. |
|
Timer |
Time spent to migrate a process instance. |
|
Timer |
Time spent to validate a migration of a process instance. |
|
Timer |
Time spent to create a process instance. |
|
Timer |
Time spent to delete a process instance. |
|
Timer |
Time spent to query process instances. |
|
Timer |
Time spent to read a process instance. |
|
Timer |
Time spent to query task definitions. |
|
Timer |
Time spent to read a task definition. |
|
Timer |
Time spent to complete a task instance. |
|
Timer |
Time spent to query task instances. |
|
Timer |
Time spent to read a task instance. |
|
Timer |
Time spent to update a task instance. |
Prometheus metrics
Metrics accessed through the Prometheus endpoint are prepended with idm_
and use underscores between words; for example, idm_recon_target_phase_seconds
. The following table lists the Prometheus metrics available in IDM:
Prometheus metrics available in IDM
Prometheus Metric Name | Type | Description |
---|---|---|
|
Summary |
Count of all audit events generated of a given topic type. |
|
Timer |
Rate of reading response objects, to fulfill the |
|
Timer |
Rate of reading response objects, to fulfill the |
|
Timer |
Rate at which filter scripts are executed, per action. Monitors scripted filters and delegated admin. |
|
Timer |
Rate of ICF query executions with queryExpression, and time taken to perform this operation. |
|
Timer |
Rate of ICF query executions with queryFilter, and time taken to perform this operation. |
|
Timer |
Rate of ICF query executions with queryId, and time taken to perform this operation. |
|
Timer |
Rate of ICF query executions when the query type is UNKNOWN, and time taken to perform this operation. |
|
Timer |
Rate of fetch operations of relationship fields for internal objects. |
|
Timer |
Query rate on relationship values for internal objects. |
|
Timer |
Rate of validate operations of relationship fields for internal objects. |
|
Timer |
Rate of script executions on internal objects. |
|
Timer |
Rate of operations on internal objects. |
|
Timer |
Rate of responses requiring field augmentation. When the repository is unable to retrieve all the data in a single call, IDM performs additional read operations to complete (augment) the missing data. |
|
Timer |
Rate of fetches of relationship fields of a managed object. |
|
Timer |
Rate of queries to get relationship values for a resource on a managed object. |
|
Timer |
Rate of validations of relationship fields of a managed object. |
|
Timer |
Rate of executions of a script on a managed object. |
|
Timer |
Latency of enforcing temporal constraints on role objects during object creation. |
|
Timer |
Latency of enforcing temporal constraints on role objects during object deletion. |
|
Timer |
Latency of enforcing temporal constraints on role objects during object update. |
|
Timer |
Latency of enforcing temporal constraints on relationship grants during edge creation. |
|
Timer |
Latency of enforcing temporal constraints on relationship grants during edge deletion. |
|
Timer |
Latency of enforcing temporal constraints on relationship grants during edge update. |
|
Timer |
Rate of reads on relationship endpoint edges for validation. |
|
Timer |
Rate of operations on a managed object. |
|
Timer |
Time spent in filter which maps non-nullable, null-valued array fields to an empty array. This filter is traversed for all repo access relating to internal and managed objects. |
|
Timer |
Rate of merge operations after source and/or target objects have been retrieved during a merged query of recon association entries. |
|
Timer |
Rate of individual paged recon association entry queries during a merged query. More than one page of entries might be requested to build a single page of merged results. |
|
Timer |
Rate of source object retrieval via query when merging source objects to recon association entries. |
|
Timer |
Rate of target object retrieval via query when merging target objects to recon association entries. |
|
Timer |
The time taken to persist association data. The operation can be |
|
Timer |
Rate of executions of the id query phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of executions of a full reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of pagination executions of the source phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of executions of the source phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of executions of the target phase of a reconciliation, and time taken to perform this operation. |
|
Timer |
Rate of filtered queries (using native query expressions) on the relationship table. This metric measures the time spent making the query (in ms), and the number of times the query is invoked. |
|
Timer |
Rate of filtered queries (using the |
|
Timer |
Rate of execution time on the JDBC database for the |
|
Timer |
Rate of execution time on the JDBC database for CRUD operations. This rate does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool. |
|
Timer |
Rate of execution time on the JDBC database for queries (either |
|
Timer |
Rate of retrievals of a repository connection. |
|
Count |
Counts the usage statistics of the |
|
Timer |
Time (ms) spent running the Edge→Vertex relationship join query on the database and collecting the result set. |
|
Timer |
Rate of relationship graph query execution times. |
|
Timer |
Rate of relationship graph query result processing times. |
|
Timer |
Rate of executions of a query with queryId at a repository level, and time taken to perform this operation. |
|
Timer |
Time (ms) spent in the various phases to retrieve relationship expanded data referenced by queried objects. |
|
Timer |
Rate of CRUDPAQ operations to a repository datasource for a generic/explicit/relationship mapped table. |
|
Timer |
Rate of actions to a repository datasource for a generic/explicit mapped table. |
|
Timer |
Rate of initiations of a CRUDPAQ operation to a repository datasource. |
|
Timer |
Rate of actions over the router, and time taken to perform this operation. |
|
Timer |
Rate of creates over the router, and time taken to perform this operation. |
|
Timer |
Rate of deletes over the router, and time taken to perform this operation. |
|
Timer |
Rate of patches over the router, and time taken to perform this operation. |
|
Timer |
Rate of queries with queryExpression completed over the router, and time taken to perform this operation. |
|
Timer |
Rate of queries with queryFilter completed over the router, and time taken to perform this operation. |
|
Timer |
Rate of reads over the router, and time taken to perform this operation. |
|
Timer |
Rate of updates over the router, and time taken to perform this operation. |
|
Timer |
Time spent storing scheduled jobs in the repository. |
|
Timer |
Execution rate of scheduler requests. |
|
Timer |
Rate of calls to a script and time taken to complete. |
|
Summary |
Count of all successful user self-service password resets. |
|
Summary |
Count of all successful user self-service registrations by registration type and provider. |
|
Summary |
Count of all successful user self-service registrations by registration type. |
|
Timer |
Rate of requests to create an object on the target, and the time taken to perform this operation. |
|
Timer |
Rate of requests to delete an object on the target, and the time taken to perform this operation. |
|
Timer |
Rate of configurations applied to a mapping. |
|
Timer |
Rate of acquisition of queued synchronization events from the queue. |
|
Timer |
Rate of deletion of synchronization events from the queue. |
|
Timer |
Rate at which queued synchronization operations are executed. |
|
Summary |
Number of queued synchronization operations that failed. |
|
Timer |
The latency involved in polling for synchronization events. |
|
Summary |
Number of queued synchronization events that were acquired by another node in the cluster. |
|
Summary |
Number of queued synchronization events that were rejected because the backing thread-pool queue was at full capacity and the thread-pool had already allocated its maximum configured number of threads. |
|
Timer |
Times the release of queued synchronization events after a failure and before exceeding the retry count. |
|
Timer |
Rate at which queued synchronization events are released. |
|
Timer |
Rate of insertion of synchronization events into the queue. |
|
Timer |
Rate of reads of an object. |
|
Timer |
Rate of assessments of a synchronization situation. |
|
Timer |
Rate of correlations between a target and a given source, and time taken to perform this operation. |
|
Timer |
Rate of determinations done on a synchronization action based on its current situation. |
|
Timer |
Rate of completions of an action performed on a synchronization operation. |
|
Timer |
Rate of assessments of a target situation. |
|
Timer |
Rate of determinations done on a target action based on its current situation. |
|
Timer |
Rate of completions of an action performed on a target sync operation. |
|
Timer |
Rate of requests to update an object on the target, and the time taken to perform this operation. |
|
Summary |
Count of all successful logins by user type. |
|
Summary |
Count of all successful logins by user type and provider. |
|
Summary |
Number of 404 responses encountered when querying the |
|
Summary |
Number of edges skipped due to an unsatisfied temporal constraint on either the edge or the referred-to vertex. Encountered when querying the resource collection and relationship field at the traversal_depthX tag for the most recent X. X corresponds to the relationship field sequence. |
|
Timer |
Time spent traversing relationship fields to calculate the specified virtual properties. The managed objects linked to by the traversal relationship fields define a tree, whose root is the virtual property host. This object tree is traversed depth-first, with the traversal_depthX corresponding to the latency involved with each relationship traversal. Traversal_depth0 corresponds to the first relationship field traversed. Because the tree is traversed depth-first, traversal_depthX will subsume all the traversal latencies for all traversal_depth Y, where Y>X. X corresponds to the relationship field sequence. |
Prometheus JVM metrics available in IDM
These metrics depend on the JVM version and configuration. In particular, garbage-collector-related metrics depend on the garbage collector that the server uses. The garbage-collector metric names are unstable, and can change even in a minor JVM release. |
Prometheus Metric Name | Type | Unit | Description |
---|---|---|---|
|
Gauge |
Count |
Number of processors available to the JVM. For more information, see Runtime. |
|
Gauge |
Count |
Number of classes loaded since the Java virtual machine started. For more information, see ClassLoadingMXBean. |
|
Gauge |
Count |
Number of classes unloaded since the Java virtual machine started. For more information, see ClassLoadingMXBean. |
|
Gauge |
Bytes |
For more information, see Runtime. |
|
Gauge |
Count |
For each garbage collector in the JVM. For more information, see GarbageCollectorMXBean. |
|
Gauge |
Milliseconds |
|
|
Gauge |
Count |
|
|
Gauge |
Milliseconds |
|
|
Gauge |
Bytes |
For more information, see Runtime. |
|
Gauge |
Bytes |
Amount of heap memory committed for the JVM to use. For more information, see MemoryMXBean. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Maximum amount of heap memory available to the JVM. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Amount of heap memory used by the JVM. |
|
Gauge |
Bytes |
Amount of non-heap memory committed for the JVM to use. |
|
Gauge |
Bytes |
Amount of non-heap memory the JVM initially requested from the operating system. |
|
Gauge |
Bytes |
Maximum amount of non-heap memory available to the JVM. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Amount of non-heap memory used by the JVM. |
|
Gauge |
Bytes |
For each pool. For more information, see MemoryPoolMXBean. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
Amount of memory that is committed for the JVM to use. For more information, see MemoryMXBean. |
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Bytes |
|
|
Gauge |
Count |
For more information, see ThreadMXBean. |
|
Gauge |
Count |
Number of live threads including both daemon and non-daemon threads. |
|
Gauge |
Count |
Number of live daemon threads. |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Count |
Number of threads in the |
|
Gauge |
Bytes |
For more information, see totalMemory(). |
Prometheus workflow metrics available in IDM
Prometheus Metric Name | Type | Description |
---|---|---|
|
Timer |
Time spent invoking a message event. |
|
Timer |
Time spent invoking a signal event. |
|
Timer |
Time spent triggering an execution. |
|
Timer |
Time spent querying executions. |
|
Timer |
Time spent forcing synchronous execution of a job. |
|
Timer |
Time spent displaying the stacktrace for a job that triggered an exception. |
|
Timer |
Time spent deleting a job. |
|
Timer |
Time spent querying jobs. |
|
Timer |
Time spent reading a single job. |
|
Timer |
Time spent to execute dead-letter job. |
|
Timer |
Time spent to retrieve the stacktrace for a dead-letter job. |
|
Timer |
Time spent to delete a dead letter job. |
|
Timer |
Time spent to query dead letter jobs. |
|
Timer |
Time spent to read a dead letter job. |
|
Timer |
Time spent to deploy a model. |
|
Timer |
Time spent to list model deployments. |
|
Timer |
Time spent to validate BPMN content. |
|
Timer |
Time spent to create a model. |
|
Timer |
Time spent to delete a model. |
|
Timer |
Time spent to query models. |
|
Timer |
Time spent to read a model. |
|
Timer |
Time spent to update a model. |
|
Timer |
Time spent to delete a process definition. |
|
Timer |
Time spent to query process definitions. |
|
Timer |
Time spent to read a process definition. |
|
Timer |
Time spent to migrate a process instance. |
|
Timer |
Time spent to validate a migration of a process instance. |
|
Timer |
Time spent to create a process instance. |
|
Timer |
Time spent to delete a process instance. |
|
Timer |
Time spent to query process instances. |
|
Timer |
Time spent to read a process instance. |
|
Timer |
Time spent to query task definitions. |
|
Timer |
Time spent to read a task definition. |
|
Timer |
Time spent to complete a task instance. |
|
Timer |
Time spent to query task instances. |
|
Timer |
Time spent to read a task instance. |
|
Timer |
Time spent to update a task instance. |