PingIDM 7.5.0

Monitoring and metrics

Configure PingIDM server logs and monitoring metrics.

ForgeRock Identity Platform™ serves as the basis for our simple and comprehensive Identity and Access Management solution. We help our customers deepen their relationships with their customers, and improve the productivity and connectivity of their employees and partners. For more information about ForgeRock and about the platform, refer to https://www.forgerock.com.

The ForgeRock Common REST API works across the platform to provide common ways to access web resources and collections of resources.

Server logs

Server logging is not the same as auditing. Auditing logs activity on the IDM system, such as access, and synchronization. Server logging records information about the internal workings of IDM, like system messages, error reporting, service loading, or startup and shutdown messaging.

Configure server logging in your project’s conf/logging.properties file. Changes to logging settings require a server restart before they take effect. Alternatively, use JMX via jconsole to change the logging settings. In this case, changes take effect without restarting the server.

Log message handlers

The way IDM logs messages is set in the handlers property in the logging.properties file. This property has the following value by default:

handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler

The default handlers are:

  • FileHandler writes formatted log records to a single file or to a set of rotating log files. By default, log files are written to logs/openidm*.log files.

  • ConsoleHandler writes formatted logs to System.err.

Additional log message handlers are listed in the logging.properties file.

Log message format

IDM supports the two default log formatters included with Java. These are set in the conf/logging.properties file:

  • java.util.logging.SimpleFormatter.format outputs a text log file that is human-readable. This is the default formatter.

  • java.util.logging.XMLFormatter outputs logs as XML, for use in logging software that can read XML logs.

IDM extends the Java SimpleFormatter with the following formatting options:

org.forgerock.openidm.logger.SanitizedThreadIdLogFormatter

This is the default formatter for console and file logging. It extends the SimpleFormatter to include the thread ID of the thread that generated each message. The thread ID helps with debugging when reviewing the logs.

In the following example log excerpt, the thread ID is [19]:

[19] May 23, 2018 10:30:26.959 AM org.forgerock.openidm.repo.opendj.impl.Activator start
INFO: Registered bootstrap repository service
[19] May 23, 2018 10:30:26.960 AM org.forgerock.openidm.repo.opendj.impl.Activator start
INFO: DS bundle started
The SanitizedThreadIdLogFormatter also encodes all control characters (such as newline characters) using URL-encoding, to protect against log forgery. Control characters in stack traces are not encoded.
org.forgerock.openidm.logger.ThreadIdLogFormatter

Similar to the SanitizedThreadIdLogFormatter, but does not encode control characters. If you do not want to encode control characters in file and console log messages, edit the file and console handlers in conf/logging.properties as follows:

java.util.logging.FileHandler.formatter = org.forgerock.openidm.logger.ThreadIdLogFormatter

java.util.logging.ConsoleHandler.formatter = org.forgerock.openidm.logger.ThreadIdLogFormatter

The SimpleFormatter (and, by extension, the SanitizedThreadIdLogFormatter and ThreadIdLogFormatter) lets you customize what information to include in log messages, and how this information is laid out. By default, log messages include the date, time (down to the millisecond), log level, source of the message, and the message sent (including exceptions). To change the defaults, adjust the value of java.util.logging.SimpleFormatter.format in your conf/logging.properties file. For more information on how to customize the log message format, refer to the related Java documentation.

Logging level

By default, IDM logs messages at the INFO level. This logging level is specified with the following global property in conf/logging.properties:

.level=INFO

You can specify different separate logging levels for individual server features which override the global logging level. Set the log level, per package to one of the following:

SEVERE (highest value)
WARNING
INFO
CONFIG
FINE
FINER
FINEST (lowest value)

For example, the following setting decreases the messages logged by the embedded PostgreSQL database:

# reduce the logging of embedded postgres since it is very verbose
ru.yandex.qatools.embed.postgresql.level = SEVERE

Set the log level to OFF to disable logging completely (Disable Logs), or to ALL to capture all possible log messages.

If you use logger functions in your JavaScript scripts, set the log level for the scripts as follows:

org.forgerock.openidm.script.javascript.JavaScript.level=level

You can override the log level settings, per script, with the following setting:

org.forgerock.openidm.script.javascript.JavaScript.script-name.level=level

For more information about using logger functions in scripts, refer to Log Functions.

It is strongly recommended that you do not log messages at the FINE or FINEST levels in a production environment. Although these levels are useful for debugging issues in a test environment, they can result in accidental exposure of sensitive data. For example, a password change patch request can expose the updated password in the Jetty logs.

Log file rotation

By default, IDM rotates log files when the size reaches 5 MB, and retains up to 5 files. All system and custom log messages are also written to these files. You can modify these limits in the following properties in the logging.properties file for your project:

# Limiting size of output file in bytes:
java.util.logging.FileHandler.limit = 5242880

# Number of output files to cycle through, by appending an
# integer to the base file name:
java.util.logging.FileHandler.count = 5

There is currently no logging.properties setting for time-based rotation of server log files. However, on UNIX systems you can use the logrotate command to schedule server log rotation at a regular interval. For more information, refer to the logrotate man page.

Disable logs

If necessary, you can disable logs. For example, to disable ConsoleHandler logging, make the following changes in your project’s conf/logging.properties file before you start IDM.

Set java.util.logging.ConsoleHandler.level = OFF, and comment out other references to ConsoleHandler, as shown in the following excerpt:

# ConsoleHandler: A simple handler for writing formatted records to System.err
#handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler
handlers=java.util.logging.FileHandler
...
# --- ConsoleHandler ---
# Default: java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.level = OFF
#java.util.logging.ConsoleHandler.formatter = ...
#java.util.logging.ConsoleHandler.filter=...

Monitoring

IDM includes the following tools for monitoring metrics:

  • A Dropwizard dashboard widget, for viewing metrics within IDM.

    Widgets are deprecated and will be removed in a future release of IDM. For more information, refer to Deprecation.

  • A Prometheus endpoint, for viewing metrics through external resources such as Prometheus and Grafana.

Enable metrics

IDM does not collect metrics by default. To enable metrics collection, open conf/metrics.json and set the enabled property to true:

{
    "enabled" : true
}

After you have enabled metrics, the following command returns all collected metrics:

Request
curl \
--header "X-OpenIDM-Username: openidm-admin" \
--header "X-OpenIDM-Password: openidm-admin" \
--header "Accept-API-Version: resource=1.0" \
--request GET \
'http://localhost:8080/openidm/metrics/api?_queryFilter=true'
Show example response
Response
{
  "result": [
    {
      "_id": "jvm.memory-usage.pools.Metaspace.used",
      "value": 101709640,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.non-heap.used",
      "value": 159728512,
      "_type": "gauge"
    },
    {
      "_id": "repo.ds.get-connection",
      "count": 25,
      "max": 13.407542,
      "mean": 7.016551422258608,
      "min": 2.274208,
      "p50": 7.038666999999999,
      "p75": 8.653042,
      "p95": 12.613916999999999,
      "p98": 13.407542,
      "p99": 13.407542,
      "p999": 13.407542,
      "stddev": 3.0043480716919446,
      "m15_rate": 1.00220378348439,
      "m1_rate": 1.0294250758954837,
      "m5_rate": 1.0065021413358448,
      "mean_rate": 1.173715776010422,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 174.284168,
      "_type": "timer"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Old-Gen.committed",
      "value": 794820608,
      "_type": "gauge"
    },
    {
      "_id": "user.session.static-user",
      "m15_rate": 0.19780232116334415,
      "m1_rate": 0.17175127368841633,
      "m5_rate": 0.1935515941358193,
      "mean_rate": 0.09993098620692964,
      "units": "events/second",
      "total": 2,
      "count": 2,
      "_type": "summary"
    },
    {
      "_id": "jvm.max-memory",
      "value": 2147483648,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Compressed-Class-Space.usage",
      "value": 0.015285782516002655,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.init",
      "value": 2555904,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.non-heap.usage",
      "value": -233855696,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Old-Gen.init",
      "value": 2034237440,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.total.max",
      "value": 2147483647,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.total.committed",
      "value": 2399019008,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.heap.init",
      "value": 2147483648,
      "_type": "gauge"
    },
    {
      "_id": "repo.ds.update.cluster",
      "count": 5,
      "max": 13.490832999999999,
      "mean": 11.40983226004801,
      "min": 8.795417,
      "p50": 10.932459,
      "p75": 12.708499999999999,
      "p95": 13.490832999999999,
      "p98": 13.490832999999999,
      "p99": 13.490832999999999,
      "p999": 13.490832999999999,
      "stddev": 1.594812363576534,
      "m15_rate": 0.2011018917421949,
      "m1_rate": 0.21471253794774184,
      "m5_rate": 0.2032510706679223,
      "mean_rate": 0.23483436767444082,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 56.608459,
      "_type": "timer"
    },
    {
      "_id": "repo.ds.read.cluster",
      "count": 5,
      "max": 13.253,
      "mean": 9.663193140378318,
      "min": 6.366667,
      "p50": 10.924292,
      "p75": 11.00375,
      "p95": 13.253,
      "p98": 13.253,
      "p99": 13.253,
      "p999": 13.253,
      "stddev": 2.480672375020272,
      "m15_rate": 0.19999386134317423,
      "m1_rate": 0.1987214208736065,
      "m5_rate": 0.19994536143224584,
      "mean_rate": 0.23467002606408544,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 49.324167,
      "_type": "timer"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.init",
      "value": 2555904,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-nmethods'.usage",
      "value": 0.42355263157894735,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Compressed-Class-Space.init",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Old-Gen.used",
      "value": 137279336,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.timed_waiting.count",
      "value": 84,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Old-Gen.usage",
      "value": 0.08353511989116669,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Metaspace.init",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Survivor-Space.committed",
      "value": 52428800,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.usage",
      "value": 0.12785444714742736,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.heap.usage",
      "value": 0.5991601198911667,
      "_type": "gauge"
    },
    {
      "_id": "jvm.garbage-collector.G1-Old-Generation.count",
      "value": 4,
      "_type": "gauge"
    },
    {
      "_id": "jvm.garbage-collector.G1-Young-Generation.count",
      "value": 18,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.waiting.count",
      "value": 50,
      "_type": "gauge"
    },
    {
      "_id": "jvm.class-loading.loaded",
      "value": 22747,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.terminated.count",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "jvm.available-cpus",
      "value": 10,
      "_type": "gauge"
    },
    {
      "_id": "jvm.garbage-collector.G1-Old-Generation.time",
      "value": 360,
      "_type": "gauge"
    },
    {
      "_id": "filter.scripted.on-request.d6fc81179beaca37094a23c2fcd00aaf54bb3ef9:router:onRequest",
      "count": 2,
      "max": 21.174791,
      "mean": 16.456464351980753,
      "min": 12.961041999999999,
      "p50": 12.961041999999999,
      "p75": 21.174791,
      "p95": 21.174791,
      "p98": 21.174791,
      "p99": 21.174791,
      "p999": 21.174791,
      "stddev": 4.061101381329072,
      "m15_rate": 0.19780232116334415,
      "m1_rate": 0.17175127368841633,
      "m5_rate": 0.1935515941358193,
      "mean_rate": 0.09992547412748008,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 34.135833,
      "_type": "timer"
    },
    {
      "_id": "jvm.memory-usage.heap.committed",
      "value": 2147483648,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Metaspace.committed",
      "value": 110043136,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.committed",
      "value": 10813440,
      "_type": "gauge"
    },
    {
      "_id": "jvm.used-memory",
      "value": 2147483648,
      "_type": "gauge"
    },
    {
      "_id": "scheduler.job-store.repo.query-list.triggers",
      "count": 5,
      "max": 21.151916999999997,
      "mean": 15.297513466089498,
      "min": 8.745917,
      "p50": 15.716375,
      "p75": 16.422957999999998,
      "p95": 21.151916999999997,
      "p98": 21.151916999999997,
      "p99": 21.151916999999997,
      "p999": 21.151916999999997,
      "stddev": 3.80884629646711,
      "m15_rate": 0.39669429076432344,
      "m1_rate": 0.355760156614281,
      "m5_rate": 0.3902458849001428,
      "mean_rate": 0.2410821468791895,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 76.092959,
      "_type": "timer"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-nmethods'.committed",
      "value": 2555904,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.total.init",
      "value": 2155151360,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-nmethods'.used",
      "value": 2432384,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.non-heap.committed",
      "value": 171778048,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Survivor-Space.usage",
      "value": 1,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Eden-Space.init",
      "value": 113246208,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Metaspace.usage",
      "value": 0.9206230255320343,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Eden-Space.max",
      "value": -1,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Old-Gen.max",
      "value": 2147483648,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.total.used",
      "value": 1520570400,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.blocked.count",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Survivor-Space.used-after-gc",
      "value": 52428800,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Eden-Space.usage",
      "value": 0.8114423851732474,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.used",
      "value": 10729600,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.used",
      "value": 33729792,
      "_type": "gauge"
    },
    {
      "_id": "repo.ds.query._adhoc-filter.scheduler",
      "count": 5,
      "max": 9.139959,
      "mean": 7.781217638351263,
      "min": 6.122667,
      "p50": 7.9247499999999995,
      "p75": 8.001249999999999,
      "p95": 9.139959,
      "p98": 9.139959,
      "p99": 9.139959,
      "p999": 9.139959,
      "stddev": 0.9531334102258491,
      "m15_rate": 0.39669429076432344,
      "m1_rate": 0.355760156614281,
      "m5_rate": 0.3902458849001428,
      "mean_rate": 0.2411032736278605,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 38.649876,
      "_type": "timer"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Survivor-Space.init",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.non-heap.max",
      "value": -1,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Survivor-Space.max",
      "value": -1,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Survivor-Space.used",
      "value": 52428800,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.max",
      "value": 122908672,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.daemon.count",
      "value": 98,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Eden-Space.used-after-gc",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.new.count",
      "value": 0,
      "_type": "gauge"
    },
    {
      "_id": "repo.ds.query._adhoc-filter.cluster",
      "count": 10,
      "max": 7.115333,
      "mean": 4.415241990632845,
      "min": 2.32275,
      "p50": 4.271917,
      "p75": 5.5420419999999995,
      "p95": 7.115333,
      "p98": 7.115333,
      "p99": 7.115333,
      "p999": 7.115333,
      "stddev": 1.57203480094502,
      "m15_rate": 0.5967004294211492,
      "m1_rate": 0.5570387357406746,
      "m5_rate": 0.590300523467897,
      "mean_rate": 0.4695941183571473,
      "duration_units": "milliseconds",
      "rate_units": "calls/second",
      "total": 43.476667,
      "_type": "timer"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Eden-Space.used",
      "value": 317718528,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Compressed-Class-Space.committed",
      "value": 14024704,
      "_type": "gauge"
    },
    {
      "_id": "jvm.garbage-collector.G1-Young-Generation.time",
      "value": 465,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-nmethods'.init",
      "value": 2555904,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.count",
      "value": 180,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.max",
      "value": 122912768,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.non-heap.init",
      "value": 7667712,
      "_type": "gauge"
    },
    {
      "_id": "audit.authentication",
      "m15_rate": 0.19780232116334415,
      "m1_rate": 0.17175127368841633,
      "m5_rate": 0.1935515941358193,
      "mean_rate": 0.09988653077391328,
      "units": "events/second",
      "total": 2,
      "count": 2,
      "_type": "summary"
    },
    {
      "_id": "jvm.memory-usage.heap.used",
      "value": 507426664,
      "_type": "gauge"
    },
    {
      "_id": "jvm.class-loading.unloaded",
      "value": 16,
      "_type": "gauge"
    },
    {
      "_id": "jvm.thread-state.runnable.count",
      "value": 46,
      "_type": "gauge"
    },
    {
      "_id": "audit.access",
      "m15_rate": 0.19779007785878447,
      "m1_rate": 0.16929634497812282,
      "m5_rate": 0.1934432200964012,
      "mean_rate": 0.05002186361867778,
      "units": "events/second",
      "total": 1,
      "count": 1,
      "_type": "summary"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.committed",
      "value": 34340864,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Eden-Space.committed",
      "value": 1300234240,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Metaspace.max",
      "value": -1,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.G1-Old-Gen.used-after-gc",
      "value": 121026408,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Compressed-Class-Space.max",
      "value": 1073741824,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.heap.max",
      "value": 2147483648,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'non-nmethods'.max",
      "value": 5836800,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.usage",
      "value": 0.39190126470490216,
      "_type": "gauge"
    },
    {
      "_id": "jvm.memory-usage.pools.Compressed-Class-Space.used",
      "value": 11149728,
      "_type": "gauge"
    },
    {
      "_id": "jvm.free-used-memory",
      "value": 860110576,
      "_type": "gauge"
    }
  ],
  "resultCount": 85,
  "pagedResultsCookie": null,
  "totalPagedResultsPolicy": "EXACT",
  "totalPagedResults": 85,
  "remainingPagedResults": -1
}
Metrics are only collected after they have been triggered by activity in IDM, such as a reconciliation.

Learn more:

Dropwizard widget

Widgets are deprecated and will be removed in a future release of IDM. For more information, refer to Deprecation.

The Dropwizard widget creates a graph of metrics based on server activity and is useful for lightweight, live monitoring of IDM. The widget has the following limitations:

  • The graph created by the widget does not persist. If you reload or navigate away from the page, the graph restarts.

  • The widget only works with time-based metrics.

To add the Dropwizard widget:

  1. From the navigation bar, click Dashboards > Dashboard Name.

  2. On the Dashboard Name page, click Add Widget.

  3. In the Add Widget window, from the Select a Widget drop-down list, select Dropwizard Table with Graph.

    Add widget window with dropwizard selected

  4. To preview any metric on the graph, click Add to Graph adjacent to any metric.

  5. Click Add.

    The Dropwizard widget now displays on the dashboard.

Prometheus endpoint

This topic describes how to configure Prometheus and Grafana to collect IDM metrics. These third-party tools are not supported by ForgeRock. Refer to the Prometheus documentation.

Prometheus is a third-party tool used for gathering and processing monitoring data. Prometheus uses the openidm/metrics/prometheus endpoint to gather information. This endpoint is protected by a basic authentication filter, using the following credentials, set in the resolver/boot.properties file:

openidm.prometheus.username=username
openidm.prometheus.password=password

The Prometheus endpoint also supports secret resolution. Refer to Secret stores.

Disable Prometheus

To disable IDM’s Prometheus handler, comment out or remove openidm.prometheus.username and openidm.prometheus.password from the resolver/boot.properties file. If these properties are not set, IDM does not enable the Prometheus handler.

Configure Prometheus

  1. Download Prometheus.

  2. Create a prometheus.yml configuration file. For more information, refer to the Prometheus configuration documentation. An example prometheus.yml file:

    global:
  scrape_interval: 15s
  external_labels:
    monitor: 'my_prometheus'

# https://prometheus.io/docs/operating/configuration/#scrape_config
scrape_configs:
  - job_name: 'openidm'
    scrape_interval: 15s
    scrape_timeout: 5s
    metrics_path: 'openidm/metrics/prometheus'
    scheme: http
    basic_auth:
      username: 'prometheus'
      password: 'prometheus'
    static_configs:
      - targets: ['localhost:8080']

    This example configures Prometheus to poll the openidm/metrics/prometheus endpoint every 5 seconds (scrape_interval: 5s), receiving metrics in a plain text format (_fields: ['text'] and _mimeType: ['text/plain;version=0.0.4']). For more information about reporting formats, refer to the Prometheus documentation on Exposition Formats.

  3. Verify the configuration returns metric results:

    Request
    curl \
--user prometheus:prometheus \
--header "Accept-API-Version: resource=1.0" \
--request GET \
'http://localhost:8080/openidm/metrics/prometheus'
    Show example response
    Response
    # HELP idm_jvm_available_cpus Automatically generated
# TYPE idm_jvm_available_cpus gauge
idm_jvm_available_cpus 10.0
# HELP idm_jvm_class_loading_loaded Automatically generated
# TYPE idm_jvm_class_loading_loaded gauge
idm_jvm_class_loading_loaded 24876.0
# HELP idm_jvm_class_loading_unloaded Automatically generated
# TYPE idm_jvm_class_loading_unloaded gauge
idm_jvm_class_loading_unloaded 1.0
# HELP idm_jvm_free_used_memory_bytes Automatically generated
# TYPE idm_jvm_free_used_memory_bytes gauge
idm_jvm_free_used_memory_bytes 9.77543264E8
# HELP idm_jvm_garbage_collector_g1_old_generation_count Automatically generated
# TYPE idm_jvm_garbage_collector_g1_old_generation_count gauge
idm_jvm_garbage_collector_g1_old_generation_count 0.0
# HELP idm_jvm_garbage_collector_g1_old_generation_time Automatically generated
# TYPE idm_jvm_garbage_collector_g1_old_generation_time gauge
idm_jvm_garbage_collector_g1_old_generation_time 0.0
# HELP idm_jvm_garbage_collector_g1_young_generation_count Automatically generated
# TYPE idm_jvm_garbage_collector_g1_young_generation_count gauge
idm_jvm_garbage_collector_g1_young_generation_count 82.0
# HELP idm_jvm_garbage_collector_g1_young_generation_time Automatically generated
# TYPE idm_jvm_garbage_collector_g1_young_generation_time gauge
idm_jvm_garbage_collector_g1_young_generation_time 2127.0
# HELP idm_jvm_max_memory_bytes Automatically generated
# TYPE idm_jvm_max_memory_bytes gauge
idm_jvm_max_memory_bytes 2.147483648E9
...

  4. Start Prometheus with the prometheus.yml configuration file:

    prometheus --config.file=/path/to/prometheus.yml

  5. To confirm that Prometheus is gathering data from IDM, go to the Prometheus monitoring page (default http://localhost:9090).

    prometheus main page

Configure Grafana

Prometheus lets you monitor and process information provided by IDM. If you need deeper analytics, you can use tools such as Grafana to create customized charts and graphs based on Prometheus data. For information on installing and running Grafana, refer to the Grafana website.

You can also monitor aspects of IDM’s performance using Prometheus to plug JVM metrics into a Grafana dashboard. For more information on using metrics to observe the system under load, refer to Load testing.

Before you get started, download the Monitoring Dashboard Samples from the BackStage download site. Open monitoring.dashboard.json from the downloaded .zip file, as you’ll need it during the following procedure.

To set up a Grafana dashboard with IDM metrics using Prometheus:

  1. In a browser, go to the main Grafana page (default http://localhost:3000) and log in.

    The default username and password for Grafana is admin.

  2. To add your Prometheus installation to Grafana as a data source, click the toggle menu button grafana toggle menu, and click Connections > Data sources.

  3. On the Data sources page, click Add data source.

  4. On the Add data source page, select Prometheus.

    grafana add prometheus source

  5. Enter information and select options, as needed. The information you enter here should match the settings in the monitoring.dashboard.json file:

    1. Give your data source a name; for example, ForgeRockIDM.

    2. Set the URL (default http://localhost:9090).

    3. Enable Basic auth.

      1. Enter the User (default prometheus).

      2. Enter the Password (default prometheus).

  6. Click Save & test.

    If the test succeeds, Grafana displays Data source is working.

Create a Grafana dashboard

After Prometheus has been configured as a data source in Grafana, you can create a dashboard with IDM metrics:

  1. In Grafana, click the toggle menu button grafana toggle menu, and click Dashboards.

  2. Click New, and do one of the following:

    • Select Import.

      1. On the Import dashboard page, drag the monitoring.dashboard.json file from its location on your system to the Upload dashboard JSON file area.

      2. Enter information in the Options area, and select the Prometheus data source you previously created.

      3. Click Import.

    • Select New dashboard.

      1. Click Add visualization.

      2. Select the Prometheus data source you previously created.

      3. Configure the panel.

        For more information, refer to:

Load testing

Load testing can help you get the most out of IDM and other ForgeRock products. The benefits load testing provides include:

  • Reducing the chance that unexpected spikes in system activity will cause the system to become unstable

  • Allowing developers and system administrators to reason more accurately and be more confident in release cycle timelines

  • Providing baseline statistics which can be used to identify and investigate unexpected behavior

Load testing is a complex subject that requires knowledge of your system and a disciplined approach. There is no "one-size-fits-all" solution that applies in all circumstances. However, there are some basic principles to keep in mind while planning, executing, and evaluating load tests.

Plan tests

The first step is to determine what metrics need to be examined, what components are going to be tested, what levels of load are going to be used, and what response ranges are acceptable. Answering these questions requires:

  • Service-level Agreements (SLAs)

  • Understanding of your use case

  • Baseline knowledge of your system

SLAs provide a stationary, business-based target to aim for in testing. An example SLA appears as follows:

Service/Endpoint Sustained load Peak load Required response time

Customer auth against LDAP repo

50,000 over 16 hours

4,000 per second three times in a 16-hour period

200ms

Employee auth against AD repo

4,000 over 10 hours

100/second

400ms

Customer registration

1,000 over 24 hours

10/second

500ms

Employee password reset

10 over 24 hours

1/second

500ms
Sample SLA warnings and details:

  • Response times are between load generator and ForgeRock platform and do not account for latency between client devices and architecture.

  • IDM must support four writes and 45 read transactions per second for 12 hours using DS as the repository.

  • IDM must support 2,000 changes from HR service.

  • Measuring response times occurs after establishing 10,000 active, concurrent stateful sessions with 10,000 unique identities.

Details will vary depending on your use case and application flow, present usage patterns, full load profile, and environment. To get the most benefit, collect this information.

The system’s full load profile depends on how it is designed and used. For example, some systems have thousands of clients each using a small slice of bandwidth, while others have only a few high-bandwidth connections. Understanding these nuances helps determine an appropriate number of connections and threads of execution to use to generate a test load.

If you have trouble determining which systems and components are being used at various points during your application flow, consider modeling your application using a sequence diagram.

Understand resource usage

Understanding what resources are heavily consumed by ForgeRock products will help you with your test planning. The following chart details some products and their consumed resources:

Product Consumed resources

AM with external stores

CPU, memory

DS as a user repository

I/O, memory

DS as a token store

I/O, memory (if high token count)

IDM

I/O; CPU and memory play an important role in provisioning, sync, and user self-service

PingGateway

CPU

All of the above depends on network performance, including name resolution and proper load balancing when required.

Execute tests

When it comes to executing tests, these are the basic principles to keep in mind:

  1. Every system is different; "it depends" is the cardinal rule.

  2. Testing scenarios that don’t happen in reality gives you test results that don’t happen in reality.

  3. System performance is constrained by the scarcest resource.

One way to ensure that your tests reflect real use patterns is to begin with a load generator that creates periods of consistent use and periods of random spikes in activity. During the consistent periods, gradually add load until you exceed your SLAs and baselines. By using that data and the data from the periods of spiking activity, you can determine how your system handles spikes in activity in many different scenarios.

Your load generator should be located on separate hardware/instances from your production systems. It should have adequate resources to generate the expected load.

When testing systems with many components, begin by testing the most basic things — I/O, CPU, and memory use. IDM provides insight into these by exposing JVM Metrics.

Once you have an understanding of the basic elements of your system, introduce new components into the tests. Keep a record of each test’s environment and the components which were under test. These components may include:

  • Hardware/Hypervisor/Container platform

  • Hosting OS/VM/Container environment

  • Hosted OS

  • Java Virtual Machine (JVM)

  • Web/J2EE Container (if used to host ForgeRock AM/IG or ForgeRock AM Agent)

  • Databases, repositories, and directory servers used with ForgeRock

  • Networking, load balancers, and firewalls between instances

  • SSL, termination points, and other communications

  • Points of integration, if any

  • Other applications and services that utilize ForgeRock components

  • Load generation configuration

  • Sample data, logs from test runs, and other generated files

While there are many tools that can help you monitor your system, a thorough understanding of your system logs is the best path to understanding its behavior.

To keep your results clear and focused, only add or adjust one variable at a time.

Do not run tests designed to stress the system to its theoretical limit. The results you get from these stress tests rarely provide actionable insights.

Change the JVM heap size

Changing the JVM heap size can improve performance and reduce the time it takes to run reconciliations.

You can set the JVM heap size via the OPENIDM_OPTS environment variable. If OPENIDM_OPTS is undefined, the JVM maximum heap size defaults to 2GB. For example, to set the minimum and maximum heap sizes to 4GB, enter the following before starting IDM:

  • Unix/Linux

  • Windows

cd /path/to/openidm/
export OPENIDM_OPTS="-Xms4096m -Xmx4096m"
./startup.sh
Using OPENIDM_HOME:   /path/to/openidm
Using PROJECT_HOME:   /path/to/openidm
Using OPENIDM_OPTS:   -Xms4096m -Xmx4096m
...
OpenIDM ready
cd \path\to\openidm
set OPENIDM_OPTS=-Xms4096m -Xmx4096m
startup.bat
"Using OPENIDM_HOME:   \path\to\openidm"
"Using PROJECT_HOME:   \path\to\openidm"
"Using OPENIDM_OPTS:   -Xms4096m -Xmx4096m -Dfile.encoding=UTF-8"
...
OpenIDM ready

You can also edit the OPENIDM_OPTS values in startup.sh or startup.bat.

Metrics reference

IDM exposes a number of metrics. All metrics are available at both the openidm/metrics/api and openidm/metrics/prometheus endpoints. The actual metric names can vary, depending on the endpoint used. Also refer to Monitoring.

Metric types

Metrics are organized into the following types:

Timer

Timers provide a histogram of the duration of an event, along with a measure of the rate of occurrences. Timers can be monitored using the Dropwizard dashboard widget and the IDM Prometheus endpoint. Durations in timers are measured in milliseconds. Rates are reported in number of calls per second. The following example shows a Timer metric:

{
   "_id": "sync.source.perform-action",
   "count": 2,
   "max": 371.53391,
   "mean": 370.1752705,
   "min": 368.816631,
   "p50": 371.53391,
   "p75": 371.53391,
   "p95": 371.53391,
   "p98": 371.53391,
   "p99": 371.53391,
   "p999": 371.53391,
   "stddev": 1.3586395,
   "m15_rate": 0.393388581528647,
   "m1_rate": 0.311520313228562,
   "m5_rate": 0.3804917698002856,
   "mean_rate": 0.08572717156016606,
   "duration_units": "milliseconds",
   "rate_units": "calls/second",
   "total": 740.350541,
   "_type": "timer"
 }

Summary

Summaries are similar to Timers in that they measure a distribution of events. However, Summaries record values that aren’t units of time, such as user login counts. Summaries cannot be graphed in the Dropwizard dashboard widget, but are available through the Prometheus endpoint, and by querying the openidm/metrics/api endpoint directly. The following example shows a Summary metric:

{
  "_id": "audit.recon",
  "m15_rate": 0.786777163057294,
  "m1_rate": 0.623040626457124,
  "m5_rate": 0.7609835396005712,
  "mean_rate": 0.16977218861919927,
  "units": "events/second",
  "total": 4,
  "count": 4,
  "_type": "summary"
}

Gauge

Gauge metrics return a numerical value that can increase or decrease. The value for a gauge is calculated on request, and represents the state of the metric at that specific time. The following example shows a Gauge metric:

{
  "_id": "jvm.used-memory",
  "value": 2147483648,
  "_type": "gauge"
}

API metrics

Metrics accessed at the api endpoint (such as those consumed by the Dropwizard dashboard widget) use dot notation for their metric names; for example, recon.target-phase. The following table lists the API metrics available in IDM:

API metrics available in IDM

API Metric Name Type Description

audit.audit-topic

Summary

Count of all audit events generated of a given topic type.

field.augmentation.edge

Timer

Rate of reading response objects to fulfill the _fields requested (when the fields were not populated by the initial repo query).

field.augmentation.vertex

Timer

Rate of reading response objects to fulfill the _fields requested (when the fields were not populated by the initial repo query).

filter.filter-type.action.script-name

Timer

Rate that filter scripts are executed per action. Monitors scripted filters and delegated admin.

icf.system-identifier.objectClass.query._queryExpression

Timer

Rate of ICF query executions with queryExpression and the time taken to perform this operation.

icf.system-identifier.objectClass.query._queryFilter

Timer

Rate of ICF query executions with queryFilter and the time taken to perform this operation.

icf.system-identifier.objectClass.query._queryId.queryId

Timer

Rate of ICF query executions with queryId, and time taken to perform this operation.

icf.system-identifier.objectClass.query._UNKNOWN

Timer

Rate of ICF query executions when the query type is UNKNOWN, and time taken to perform this operation.

internal.managed-object.operation

Timer

Rate of operations on internal objects.

internal.managed-object.relationship.fetch-relationship-fields

Timer

Rate of fetch operations of relationship fields for internal objects.

internal.managed-object.relationship.get-relationship-value-for-resource

Timer

Query rate on relationship values for internal objects.

internal.managed-object.script.script-name

Timer

Rate of script executions on internal object.

internal.managed-object.relationship.validate-relationship-fields

Timer

Rate of validate operations of relationship fields for internal objects.

live-sync.system-name.object-type

Timer

Duration of live sync on a system object.

managed.field.augmentation

Timer

Rate of responses requiring field augmentation. When the repository cannot retrieve all data in a single call, IDM performs additional read operations to complete (augment) the missing data.

managed.managed-object.operation

Timer

Rate of operations on a managed object.

managed.managed-object.relationship.fetch-relationship-fields

Timer

Rate of fetches of relationship fields of a managed object.

managed.managed-object.relationship.get-relationship-value-for-resource

Timer

Rate of queries to get relationship values for a resource on a managed object.

managed.managed-object.relationship.validate-relationship-fields

Timer

Rate of validations of relationship fields of a managed object.

managed.managed-object.script.script-name

Timer

Rate of executions of a script on a managed object.

managed.object.handle-temporal-constraints-on-create

Timer

Latency of enforcing temporal constraints on role objects during object creation.

managed.object.handle-temporal-constraints-on-delete

Timer

Latency of enforcing temporal constraints on role objects during object deletion.

managed.object.handle-temporal-constraints-on-update

Timer

Latency of enforcing temporal constraints on role objects during object update.

managed.relationship.handle-temporal-constraints-on-create

Timer

Latency of enforcing temporal constraints on relationship grants during edge creation.

managed.relationship.handle-temporal-constraints-on-delete

Timer

Latency of enforcing temporal constraints on relationship grants during edge deletion.

managed.relationship.handle-temporal-constraints-on-update

Timer

Latency of enforcing temporal constraints on relationship grants during edge update.

managed.relationship.validate.read-relationship-endpoint-edges

Timer

Rate of reads on relationship endpoint edges for validation.

null_array_filter.augmentationrequestType

Timer

Time spent in filter that maps non-nullable and null-valued array fields to an empty array. This filter is traversed for all repo access relating to internal and managed objects.

recon

Timer

Rate of executions of a full reconciliation, and time taken to perform this operation.

recon-assoc-entry.merged-query.merge-results

Timer

Rate of merge operations after source and/or target objects have been retrieved during a merged query of recon association entries.

recon-assoc-entry.merged-query.page-assoc-entries

Timer

Rate of individual paged recon association entry queries during a merged query. More than one page of entries might be requested to build a single page of merged results.

recon-assoc-entry.merged-query.query-source

Timer

Rate of source object retrieval via query when merging source objects to recon association entries.

recon-assoc-entry.merged-query.query-target

Timer

Rate of target object retrieval via query when merging target objects to recon association entries.

recon.association-persistence.recon-id-operation

Timer

The time taken to persist association data. The operation can be source, target, or amendsource, depending on whether data is being produced for a source-phase or target-phase recon association, or to amend the association for a specific source.

recon.id-queries-phase

Timer

Rate of executions of the id query phase of a reconciliation, and time taken to perform this operation.

recon.source-phase

Timer

Rate of executions of the source phase of a reconciliation, and time taken to perform this operation.

recon.source-phase.page

Timer

Rate of pagination executions of the source phase of a reconciliation, and time taken to perform this operation.

recon.target-phase

Timer

Rate of executions of the target phase of a reconciliation, and time taken to perform this operation.

repo.jdbc.relationship.edge.execute.joinedToVertex

Timer

Time (ms) spent running the Edge→Vertex relationship join query on the database and collecting the result set.

repo.jdbc.relationship.execute

Timer

Rate of relationship graph query execution times.

repo.jdbc.relationship.process

Timer

Rate of relationship graph query result processing times.

repo.raw._queryId.queryId

Timer

Rate of executions of a query with queryId at a repository level and the time taken to perform this operation.

repo.repo-type.cache.objecttypes.event.resource-mapping

Count

Counts the usage statistics of the objecttypeid cache, which maps an object type to its objecttypeid. The expected count is a small number of misses (sometimes, only one) and the remainder of hits.

repo.repo-typeget-connection

Timer

Rate of retrievals of a repository connection.

repo.repo-type.operation.action_name.command.resource-mapping

Timer

Rate of actions to a repository datasource for a generic/explicit mapped table.

repo.repo-type.operation._adhoc-expression.relationship

Timer

Rate of filtered queries (using native query expressions) on the relationship table. This metric measures the time spent making the query (in ms), and the number of times the query is invoked.

repo.repo-type.operation._adhoc-filter.relationship

Timer

Rate of filtered queries (using the _queryFilter parameter) on the relationship table. This metric measures the time spent making the query (in ms), and the number of times the query is invoked.

repo.repo-type.create_properties.execute.resource-mapping

Timer

Rate of execution time on the JDBC database for the create_properties operations. This operation is performed for every generic object create when it persists the searchable properties. The rate measured here does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool.

repo.repo-type.operation.execute.resource-mapping

Timer

Rate of execution time on the JDBC database for CRUD operations. This rate does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool.

repo.repo-type.query.execute.resource-mappingqueryType.]

Timer

Rate of execution time on the JDBC database for queries (either queryFilter or queryId). This rate does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool.

repo.repo-type.operation.relationship

Timer

Rate of CRUDPAQ operations to a repository datasource for a generic/explicit/relationship mapped table.

repo.repo-type.operation.relationship.stage.origin_type

Timer

Time (ms) spent in the various phases to retrieve relationship expanded data referenced by queried objects.

repo.repo-type.operation.resource-mapping

Timer

Rate of initiations of a CRUDPAQ operation to a repository datasource.

router.path-name.action.action-type

Timer

Rate of actions over the router and the time taken to perform this operation.

router.path-name.create

Timer

Rate of creates over the router and the time taken to perform this operation.

router.path-name.delete

Timer

Rate of deletes over the router and the time taken to perform this operation.

router.path-name.patch

Timer

Rate of patches over the router and the time taken to perform this operation.

router.path-name.query.queryExpression

Timer

Rate of queries with queryExpression completed over the router and the time taken to perform this operation.

router.path-name.query.queryFilter

Timer

Rate of queries with queryFilter completed over the router and the time taken to perform this operation.

router.path-name.read

Timer

Rate of reads over the router and the time taken to perform this operation.

router.path-name.update

Timer

Rate of updates over the router and the time taken to perform this operation.

script.script-name.request-type

Timer

Rate of calls to a script and time taken to complete.

selfservice.user.password.reset

Summary

Count of all successful user self-service password resets.

selfservice.user.registration.registration-type

Summary

Count of all successful user self-service registrations by registration type.

selfservice.user.registration.registration-type.provider

Summary

Count of all successful user self-service registrations by registration type and provider.

sync.create-object

Timer

Rate of requests to create a target object, and time taken to perform the operation.

sync.delete-target

Timer

Rate of requests to delete a target object, and time taken to perform the operation.

sync.objectmapping.mapping-name

Timer

Rate of configurations applied to a mapping.

sync.queue.mapping-name.action.acquire

Timer

Rate of acquisition of queued synchronization events from the queue.

sync.queue.mapping-name.action.discard

Timer

Rate of deletion of synchronization events from the queue.

sync.queue.mapping-name.action.execution

Timer

Rate at which queued synchronization operations are executed.

sync.queue.mapping-name.action.failed

Summary

Number of queued synchronization operations that failed.

sync.queue.mapping-name.action.precondition-failed

Summary

Number of queued synchronization events acquired by another node in the cluster.

sync.queue.mapping-name.action.rejected-executions

Summary

Number of queued synchronization events rejected because the backing thread-pool queue was at full capacity and the thread-pool had already allocated its maximum configured number of threads.

sync.queue.mapping-name.action.release

Timer

Rate at which queued synchronization events are released.

sync.queue.mapping-name.action.release-for-retry

Timer

Times the release of queued synchronization events after a failure and before exceeding the retry count.

sync.queue.mapping-name.action.submit

Timer

Rate of insertion of synchronization events into the queue.

sync.queue.mapping-name.poll-pending-events

Timer

The latency involved in polling for synchronization events.

sync.raw-read-object

Timer

Rate of reads of an object.

sync.source.assess-situation

Timer

Rate of assessments of a synchronization situation.

sync.source.correlate-target

Timer

Rate of correlations between a target and a given source, and time taken to perform this operation.

sync.source.determine-action

Timer

Rate of determinations done on a synchronization action based on its current situation.

sync.source.perform-action

Timer

Rate of completions of an action performed on a synchronization operation.

sync.target.assess-situation

Timer

Rate of assessments of a target situation.

sync.target.determine-action

Timer

Rate of determinations done on a target action based on its current situation.

sync.target.perform-action

Timer

Rate of completions of an action performed on a target sync operation.

sync.update-target

Timer

Rate of requests to update an object on the target, and the time taken to perform this operation.

user.login.user-type

Summary

Count of all successful logins by user type.

user.login.user-type.provider

Summary

Count of all successful logins by user type and provider.

virtual-properties-from-relationships.not-found.virtual_properties.resource_collection_relationship_field

Summary

Number of 404 responses encountered when querying the resource_collection/relationship_field specified in the traversal_depthX tag for the most recent X.

virtual-properties-from-relationships.unsatisified-temp-constraint.virtual_properties.resource_collection_relationship_field

Summary

Number of edges skipped due to an unsatisfied temporal constraint on either the edge or the referred-to vertex. Encountered when querying the resource collection and relationship field at the traversal_depthX tag for the most recent X.

virtual-properties-from-relationships.virtual_properties.resource_collection_relationship_field

Timer

Time spent traversing relationship fields to calculate the specified virtual properties. The managed objects linked to by the traversal relationship fields define a tree whose root is the virtual property host. This object tree is traversed depth-first with the traversal_depthX corresponding to the latency involved with each relationship traversal. Traversal_depth0 corresponds to the first relationship field traversed. Because the tree is traversed depth-first, traversal_depthX subsumes all the traversal latencies for all traversal_depth Y, where Y>X.

API JVM metrics available in IDM

These metrics depend on the JVM version and configuration. In particular, garbage-collector-related metrics depend on the garbage collector that the server uses. The garbage-collector metric names are unstable and can change even in a minor JVM release.
API Metric Name Type Unit Description

jvm.available-cpus

Gauge

Count

Number of processors available to the JVM. For more information, refer to Runtime.

jvm.class-loading.loaded

Gauge

Count

Number of classes loaded since the Java virtual machine started. For more information, refer to ClassLoadingMXBean.

jvm.class-loading.unloaded

Gauge

Count

Number of classes unloaded since the Java virtual machine started. For more information, refer to ClassLoadingMXBean.

jvm.free-used-memory

Gauge

Bytes

For more information, refer to Runtime.

jvm.garbage-collector.G1-Old-Generation.count

Gauge

Count

For each garbage collector in the JVM. For more information, refer to GarbageCollectorMXBean.

jvm.garbage-collector.G1-Old-Generation.time

Gauge

Milliseconds

jvm.garbage-collector.G1-Young-Generation.count

Gauge

Count

jvm.garbage-collector.G1-Young-Generation.time

Gauge

Milliseconds

jvm.max-memory

Gauge

Bytes

For more information, refer to Runtime.

jvm.memory-usage.heap.committed

Gauge

Bytes

Amount of heap memory committed for the JVM to use. For more information, refer to MemoryMXBean.

jvm.memory-usage.heap.init

Gauge

Bytes

jvm.memory-usage.heap.max

Gauge

Bytes

Maximum amount of heap memory available to the JVM.

jvm.memory-usage.heap.usage

Gauge

Bytes

jvm.memory-usage.heap.used

Gauge

Bytes

Amount of heap memory used by the JVM.

jvm.memory-usage.non-heap.committed

Gauge

Bytes

Amount of non-heap memory committed for the JVM to use.

jvm.memory-usage.non-heap.init

Gauge

Bytes

Amount of non-heap memory the JVM initially requested from the operating system.

jvm.memory-usage.non-heap.max

Gauge

Bytes

Maximum amount of non-heap memory available to the JVM.

jvm.memory-usage.non-heap.usage

Gauge

Bytes

jvm.memory-usage.non-heap.used

Gauge

Bytes

Amount of non-heap memory used by the JVM.

jvm.memory-usage.pools.CodeHeap-'non-nmethods'.committed

Gauge

Bytes

For each pool. For more information, refer to MemoryPoolMXBean.

jvm.memory-usage.pools.CodeHeap-'non-nmethods'.init

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-nmethods'.max

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-nmethods'.usage

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-nmethods'.used

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.committed

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.init

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.max

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.usage

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'non-profiled-nmethods'.used

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.committed

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.init

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.max

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.usage

Gauge

Bytes

jvm.memory-usage.pools.CodeHeap-'profiled-nmethods'.used

Gauge

Bytes

jvm.memory-usage.pools.Compressed-Class-Space.committed

Gauge

Bytes

jvm.memory-usage.pools.Compressed-Class-Space.init

Gauge

Bytes

jvm.memory-usage.pools.Compressed-Class-Space.max

Gauge

Bytes

jvm.memory-usage.pools.Compressed-Class-Space.usage

Gauge

Bytes

jvm.memory-usage.pools.Compressed-Class-Space.used

Gauge

Bytes

jvm.memory-usage.pools.G1-Eden-Space.committed

Gauge

Bytes

jvm.memory-usage.pools.G1-Eden-Space.init

Gauge

Bytes

jvm.memory-usage.pools.G1-Eden-Space.max

Gauge

Bytes

jvm.memory-usage.pools.G1-Eden-Space.usage

Gauge

Bytes

jvm.memory-usage.pools.G1-Eden-Space.used

Gauge

Bytes

jvm.memory-usage.pools.G1-Eden-Space.used-after-gc

Gauge

Bytes

jvm.memory-usage.pools.G1-Old-Gen.committed

Gauge

Bytes

jvm.memory-usage.pools.G1-Old-Gen.init

Gauge

Bytes

jvm.memory-usage.pools.G1-Old-Gen.max

Gauge

Bytes

jvm.memory-usage.pools.G1-Old-Gen.usage

Gauge

Bytes

jvm.memory-usage.pools.G1-Old-Gen.used

Gauge

Bytes

jvm.memory-usage.pools.G1-Old-Gen.used-after-gc

Gauge

Bytes

jvm.memory-usage.pools.G1-Survivor-Space.committed

Gauge

Bytes

jvm.memory-usage.pools.G1-Survivor-Space.init

Gauge

Bytes

jvm.memory-usage.pools.G1-Survivor-Space.max

Gauge

Bytes

jvm.memory-usage.pools.G1-Survivor-Space.usage

Gauge

Bytes

jvm.memory-usage.pools.G1-Survivor-Space.used

Gauge

Bytes

jvm.memory-usage.pools.G1-Survivor-Space.used-after-gc

Gauge

Bytes

jvm.memory-usage.pools.Metaspace.committed

Gauge

Bytes

jvm.memory-usage.pools.Metaspace.init

Gauge

Bytes

jvm.memory-usage.pools.Metaspace.max

Gauge

Bytes

jvm.memory-usage.pools.Metaspace.usage

Gauge

Bytes

jvm.memory-usage.pools.Metaspace.used

Gauge

Bytes

jvm.memory-usage.total.committed

Gauge

Bytes

Amount of memory that is committed for the JVM to use. For more information, refer to MemoryMXBean.

jvm.memory-usage.total.init

Gauge

Bytes

jvm.memory-usage.total.max

Gauge

Bytes

jvm.memory-usage.total.used

Gauge

Bytes

jvm.thread-state.blocked.count

Gauge

Count

For more information, refer to ThreadMXBean.

jvm.thread-state.count

Gauge

Count

Number of live threads including both daemon and non-daemon threads.

jvm.thread-state.daemon.count

Gauge

Count

Number of live daemon threads.

jvm.thread-state.new.count

Gauge

Count

Number of threads in the NEW state.

jvm.thread-state.runnable.count

Gauge

Count

Number of threads in the RUNNABLE state.

jvm.thread-state.terminated.count

Gauge

Count

Number of threads in the TERMINATED state.

jvm.thread-state.timed_waiting.count

Gauge

Count

Number of threads in the TIMED_WAITING state.

jvm.thread-state.waiting.count

Gauge

Count

Number of threads in the WAITING state.

jvm.used-memory

Gauge

Bytes

For more information, refer to totalMemory().

API scheduler metrics available in IDM

For example requests, refer to Scheduler metrics.

API Metric Name Type Description

scheduler.job.job-group.job-name.completed

Summary

A summary of completed jobs for the specified job-group and job-name.

scheduler.job.job-group.job-name.executed

Timer

Time spent on executed jobs for the specified job-group and job-name.

scheduler.job-store.repo.operation.scheduler-object

Timer

Time spent storing scheduled jobs in the repository for the specified operation and scheduler-object.

scheduler.trigger.acquired.success

Summary

A summary of successfully acquired jobs.

scheduler.trigger.acquired.timeout

Summary

A summary of acquired jobs that time out.

scheduler.trigger.fired

Summary

A summary of fired schedule triggers.

scheduler.trigger.misfired

Summary

A summary of misfired schedule triggers.

scheduler.trigger.recovered

Timer

Time spent on recovered triggers.

scheduler.type.operation

Timer

Execution rate of scheduler requests for the specified type and operation.

API workflow metrics available in IDM

API Metric Name Type Description

workflow.execution.action.message

Timer

Time spent invoking a message event.

workflow.execution.action.signal

Timer

Time spent invoking a signal event.

workflow.execution.action.trigger

Timer

Time spent triggering an execution.

workflow.execution.query

Timer

Time spent querying executions.

workflow.job.action.execute

Timer

Time spent forcing synchronous execution of a job.

workflow.job.action.stacktrace

Timer

Time spent displaying the stacktrace for a job that triggered an exception.

workflow.job.delete

Timer

Time spent deleting a job.

workflow.job.query

Timer

Time spent querying jobs.

workflow.job.read

Timer

Time spent reading a single job.

workflow.jobdeadletter.action.execute

Timer

Time spent to execute dead-letter job.

workflow.jobdeadletter.action.stacktrace

Timer

Time spent to retrieve the stacktrace for a dead-letter job.

workflow.jobdeadletter.delete

Timer

Time spent to delete a dead letter job.

workflow.jobdeadletter.query

Timer

Time spent to query dead letter jobs.

workflow.jobdeadletter.read

Timer

Time spent to read a dead letter job.

workflow.model.action.deploy

Timer

Time spent to deploy a model.

workflow.model.action.list_deployments

Timer

Time spent to list model deployments.

workflow.model.action.validate_bpmn

Timer

Time spent to validate BPMN content.

workflow.model.create

Timer

Time spent to create a model.

workflow.model.delete

Timer

Time spent to delete a model.

workflow.model.query

Timer

Time spent to query models.

workflow.model.read

Timer

Time spent to read a model.

workflow.model.update

Timer

Time spent to update a model.

workflow.processdefinition.delete

Timer

Time spent to delete a process definition.

workflow.processdefinition.query

Timer

Time spent to query process definitions.

workflow.processdefinition.read

Timer

Time spent to read a process definition.

workflow.processinstance.action.migrate

Timer

Time spent to migrate a process instance.

workflow.processinstance.action.validateMigration

Timer

Time spent to validate a migration of a process instance.

workflow.processinstance.create

Timer

Time spent to create a process instance.

workflow.processinstance.delete

Timer

Time spent to delete a process instance.

workflow.processinstance.query

Timer

Time spent to query process instances.

workflow.processinstance.read

Timer

Time spent to read a process instance.

workflow.taskdefinition.query

Timer

Time spent to query task definitions.

workflow.taskdefinition.read

Timer

Time spent to read a task definition.

workflow.taskinstance.action.complete

Timer

Time spent to complete a task instance.

workflow.taskinstance.query

Timer

Time spent to query task instances.

workflow.taskinstance.read

Timer

Time spent to read a task instance.

workflow.taskinstance.update

Timer

Time spent to update a task instance.

Prometheus metrics

Metrics accessed through the Prometheus endpoint are prepended with idm_ and use underscores between words; for example, idm_recon_target_phase_seconds. The following table lists the Prometheus metrics available in IDM:

Prometheus metrics available in IDM

Prometheus Metric Name Type Description

idm_audit{audit_topic=audit-topic}

Summary

Count of all audit events generated of a given topic type.

idm_field_augmentation{origin-type=edge}

Timer

Rate of reading response objects, to fulfill the _fields requested (when the fields were not populated by the initial repo query).

idm_field_augmentation{origin-type=vertex}

Timer

Rate of reading response objects, to fulfill the _fields requested (when the fields were not populated by the initial repo query).

idm_filter_seconds{action=action,filter_type=filter-type,script_name=script-name}

Timer

Rate at which filter scripts are executed, per action. Monitors scripted filters and delegated admin.

idm_icf_system-identifier_objectClass_query__queryExpression_seconds

Timer

Rate of ICF query executions with queryExpression, and time taken to perform this operation.

idm_icf_system-identifier_objectClass_query__queryFilter_seconds

Timer

Rate of ICF query executions with queryFilter, and time taken to perform this operation.

idm_icf_system-identifier_objectClass_query__queryId_queryId_seconds

Timer

Rate of ICF query executions with queryId, and time taken to perform this operation.

idm_icf_system-identifier_objectClass_query__UNKNOWN_seconds

Timer

Rate of ICF query executions when the query type is UNKNOWN, and time taken to perform this operation.

idm_internal_managed-object_relationship_fetch_relationship_fields_seconds

Timer

Rate of fetch operations of relationship fields for internal objects.

idm_internal_managed-object_relationship_get_relationship_value_for_resource_seconds

Timer

Query rate on relationship values for internal objects.

idm_internal_managed-object_relationship_validate_relationship_fields_seconds

Timer

Rate of validate operations of relationship fields for internal objects.

idm_internal_managed-object_script_script-name_seconds

Timer

Rate of script executions on internal objects.

idm_internal_seconds{managed_object=managed-object,operation=operation}

Timer

Rate of operations on internal objects.

idm_live-sync{sytem-name=system-name,object-type=object-type}

Timer

Duration of live sync on a system object.

idm_managed_field_augmentation_seconds

Timer

Rate of responses requiring field augmentation. When the repository is unable to retrieve all the data in a single call, IDM performs additional read operations to complete (augment) the missing data.

idm_managed_managed-object_relationship_fetch_relationship_fields_seconds

Timer

Rate of fetches of relationship fields of a managed object.

idm_managed_managed-object_relationship_get_relationship_value_for_resource_seconds

Timer

Rate of queries to get relationship values for a resource on a managed object.

idm_managed_managed-object_relationship_validate_relationship_fields_seconds

Timer

Rate of validations of relationship fields of a managed object.

idm_managed_managed-objectscriptscript-name_seconds

Timer

Rate of executions of a script on a managed object.

idm_managed_object_handle_temporal_constraints_on_create

Timer

Latency of enforcing temporal constraints on role objects during object creation.

idm_managed_object_handle_temporal_constraints_on_delete

Timer

Latency of enforcing temporal constraints on role objects during object deletion.

idm_managed_object_handle_temporal_constraints_on_update

Timer

Latency of enforcing temporal constraints on role objects during object update.

idm_managed_relationship_handle_temporal_constraints_on_create

Timer

Latency of enforcing temporal constraints on relationship grants during edge creation.

idm_managed_relationship_handle_temporal_constraints_on_delete

Timer

Latency of enforcing temporal constraints on relationship grants during edge deletion.

idm_managed_relationship_handle_temporal_constraints_on_update

Timer

Latency of enforcing temporal constraints on relationship grants during edge update.

idm_managed_relationship_validate_read_relationship_endpoint_edges_seconds

Timer

Rate of reads on relationship endpoint edges for validation.

idm_managed_seconds{managed_object=managed-object,operation=operation}

Timer

Rate of operations on a managed object.

idm_null_array_filter.augmentationrequestType

Timer

Time spent in filter which maps non-nullable, null-valued array fields to an empty array. This filter is traversed for all repo access relating to internal and managed objects.

idm_recon-assoc-entry_merged-query_merge-results

Timer

Rate of merge operations after source and/or target objects have been retrieved during a merged query of recon association entries.

idm_recon-assoc-entry_merged-query_page-assoc-entries

Timer

Rate of individual paged recon association entry queries during a merged query. More than one page of entries might be requested to build a single page of merged results.

idm_recon-assoc-entry_merged-query_query-source

Timer

Rate of source object retrieval via query when merging source objects to recon association entries.

idm_recon-assoc-entry_merged-query_query-target

Timer

Rate of target object retrieval via query when merging target objects to recon association entries.

idm_recon_association-persistence{recon-id=reconId,operation=operation}

Timer

The time taken to persist association data. The operation can be source, target, or amendsource, depending on whether data is being produced for a source-phase or target-phase recon association, or to amend the association for a specific source.

idm_recon_id_queries_phase_seconds

Timer

Rate of executions of the id query phase of a reconciliation, and time taken to perform this operation.

idm_recon_seconds

Timer

Rate of executions of a full reconciliation, and time taken to perform this operation.

idm_recon_source_phase_page_seconds

Timer

Rate of pagination executions of the source phase of a reconciliation, and time taken to perform this operation.

idm_recon_source_phase_seconds

Timer

Rate of executions of the source phase of a reconciliation, and time taken to perform this operation.

idm_recon_target_phase_seconds

Timer

Rate of executions of the target phase of a reconciliation, and time taken to perform this operation.

idm_repo_adhoc-expression_relationship_seconds{operation=operation,repo_type=repo-type}

Timer

Rate of filtered queries (using native query expressions) on the relationship table. This metric measures the time spent making the query (in ms), and the number of times the query is invoked.

idm_repo_adhoc-filter_relationship_seconds{operation=operation,repo_type=repo-type}

Timer

Rate of filtered queries (using the _queryFilter parameter) on the relationship table. This metric measures the time spent making the query (in ms), and the number of times the query is invoked.

idm_repo_execute_seconds{operation=create_properties,repo_type=repo-type,resource_mapping=resource-mapping}

Timer

Rate of execution time on the JDBC database for the create_properties operations. This operation is performed for every generic object create when it persists the searchable properties. The rate measured here does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool.

idm_repo_execute_seconds{operation=operation,repo_type=repo-type,resource_mapping=resource-mapping}

Timer

Rate of execution time on the JDBC database for CRUD operations. This rate does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool.

idm_repo_execute_seconds{operation="query",queryType=queryFilter|queryId,repo_type=repo-type,resource_mapping=resource-mapping}

Timer

Rate of execution time on the JDBC database for queries (either queryFilter or queryId). This rate does not include the time taken to obtain a connection to the database from the connection pool. The physical connections to the database have already been established inside the connection pool.

idm_repo_get_connection_seconds{repo_type=repo-type}

Timer

Rate of retrievals of a repository connection.

idm_repo_jdbc_cache_objecttypes_count{event="hit|miss",type=resource-mapping

Count

Counts the usage statistics of the objecttypeid cache, which maps an object type to its objecttypeid. The expected count is a small number of misses (sometimes, only one) and the remainder of hits.

idm_repo_jdbc_relationship_edge_execute_seconds{joinedToVertex=joinedToVertex>

Timer

Time (ms) spent running the Edge→Vertex relationship join query on the database and collecting the result set.

idm_repo_jdbc_relationship_execute_seconds

Timer

Rate of relationship graph query execution times.

idm_repo_jdbc_relationship_process_seconds

Timer

Rate of relationship graph query result processing times.

idm_repo_raw__queryid_credential_queryId_seconds

Timer

Rate of executions of a query with queryId at a repository level, and time taken to perform this operation.

idm_repo_relationship_count{operation=operation,origin_type=origin_type,repo_type=repo_type,stage=stage}

Timer

Time (ms) spent in the various phases to retrieve relationship expanded data referenced by queried objects.

idm_repo_relationship_seconds{operation=operation,repo_type=repo-type}

Timer

Rate of CRUDPAQ operations to a repository datasource for a generic/explicit/relationship mapped table.

idm_repo_seconds{action_name=action-name,command=command,operation=operation,repo_type=repo-type,resource_mapping=resource-mapping}

Timer

Rate of actions to a repository datasource for a generic/explicit mapped table.

idm_repo_seconds{operation=operation,repo_type=repo-type,resource_mapping=resource-mapping}

Timer

Rate of initiations of a CRUDPAQ operation to a repository datasource.

idm_router_path-nameactionaction-type_seconds

Timer

Rate of actions over the router, and time taken to perform this operation.

idm_router_path-name_create_seconds

Timer

Rate of creates over the router, and time taken to perform this operation.

idm_router_path-name_delete_seconds

Timer

Rate of deletes over the router, and time taken to perform this operation.

idm_router_path-name_patch_seconds

Timer

Rate of patches over the router, and time taken to perform this operation.

idm_router_path-name_query_queryExpression_seconds

Timer

Rate of queries with queryExpression completed over the router, and time taken to perform this operation.

idm_router_path-name_query_queryFilter_seconds

Timer

Rate of queries with queryFilter completed over the router, and time taken to perform this operation.

idm_router_path-name_read_seconds

Timer

Rate of reads over the router, and time taken to perform this operation.

idm_router_path-name_update_seconds

Timer

Rate of updates over the router, and time taken to perform this operation.

idm_script_script-name_request-type

Timer

Rate of calls to a script and time taken to complete.

idm_selfservice_user_password_reset

Summary

Count of all successful user self-service password resets.

idm_selfservice_user_registration{provider=provider,reg_type=registration-type}

Summary

Count of all successful user self-service registrations by registration type and provider.

idm_selfservice_user_registration{reg_type=registration-type}

Summary

Count of all successful user self-service registrations by registration type.

idm_sync_create_object_seconds

Timer

Rate of requests to create an object on the target, and the time taken to perform this operation.

idm_sync_delete_target_seconds

Timer

Rate of requests to delete an object on the target, and the time taken to perform this operation.

idm_sync_objectmapping_seconds{mapping_name=mapping-name}

Timer

Rate of configurations applied to a mapping.

idm_sync_queue_acquire{mapping_name=mapping-name, action=action}

Timer

Rate of acquisition of queued synchronization events from the queue.

idm_sync_queue_discard{mapping_name=mapping-name, action=action}

Timer

Rate of deletion of synchronization events from the queue.

idm_sync_queue_execution{mapping_name=mapping-name, action=action}

Timer

Rate at which queued synchronization operations are executed.

idm_sync_queue_failed{mapping_name=mapping-name, action=action}

Summary

Number of queued synchronization operations that failed.

idm_sync_queue_poll_pending_events{mapping_name=mapping-name}

Timer

The latency involved in polling for synchronization events.

idm_sync_queue_precondition_failed{mapping_name=mapping-name, action=action}

Summary

Number of queued synchronization events that were acquired by another node in the cluster.

idm_sync_queue_rejected_executions{mapping_name=mapping-name, action=action}

Summary

Number of queued synchronization events that were rejected because the backing thread-pool queue was at full capacity and the thread-pool had already allocated its maximum configured number of threads.

idm_sync_queue_release_for_retry{mapping_name=mapping-name, action=action}

Timer

Times the release of queued synchronization events after a failure and before exceeding the retry count.

idm_sync_queue_release{mapping_name=mapping-name, action=action}

Timer

Rate at which queued synchronization events are released.

idm_sync_queue_submit{mapping_name=mapping-name, action=action}

Timer

Rate of insertion of synchronization events into the queue.

idm_sync_raw_read_object_seconds

Timer

Rate of reads of an object.

idm_sync_source_assess_situation_seconds

Timer

Rate of assessments of a synchronization situation.

idm_sync_source_correlate_target_seconds

Timer

Rate of correlations between a target and a given source, and time taken to perform this operation.

idm_sync_source_determine_action_seconds

Timer

Rate of determinations done on a synchronization action based on its current situation.

idm_sync_source_perform_action_seconds

Timer

Rate of completions of an action performed on a synchronization operation.

idm_sync_target_assess_situation_seconds

Timer

Rate of assessments of a target situation.

idm_sync_target_determine_action_seconds

Timer

Rate of determinations done on a target action based on its current situation.

idm_sync_target_perform_action_seconds

Timer

Rate of completions of an action performed on a target sync operation.

idm_sync_update_target_seconds

Timer

Rate of requests to update an object on the target, and the time taken to perform this operation.

idm_user_login{user_type=user-type}

Summary

Count of all successful logins by user type.

idm_user_login_total{provider=provider,user_type=user-type}

Summary

Count of all successful logins by user type and provider.

idm_virtual_properties_from_relationships{virtual_properties=calculated-virtual-properties, traversal_depthX=traversal-origin-resource-collection and traversal relationship,not_found}

Summary

Number of 404 responses encountered when querying the resource_collection/relationship_field specified in the traversal_depthX tag for the most recent X. X corresponds to the relationship field sequence.

idm_virtual_properties_from_relationships{virtual_properties=calculated-virtual-properties, traversal_depthX=traversal-origin-resource-collection and traversal relationship,unsatisfied_temp_constraint}

Summary

Number of edges skipped due to an unsatisfied temporal constraint on either the edge or the referred-to vertex. Encountered when querying the resource collection and relationship field at the traversal_depthX tag for the most recent X. X corresponds to the relationship field sequence.

idm_virtual_properties_from_relationships{virtual_properties=calculated-virtual-properties, traversal_depthX=traversal-origin-resource-collection and traversal relationship}

Timer

Time spent traversing relationship fields to calculate the specified virtual properties. The managed objects linked to by the traversal relationship fields define a tree, whose root is the virtual property host. This object tree is traversed depth-first, with the traversal_depthX corresponding to the latency involved with each relationship traversal. Traversal_depth0 corresponds to the first relationship field traversed. Because the tree is traversed depth-first, traversal_depthX will subsume all the traversal latencies for all traversal_depth Y, where Y>X. X corresponds to the relationship field sequence.

Prometheus JVM metrics available in IDM

These metrics depend on the JVM version and configuration. In particular, garbage-collector-related metrics depend on the garbage collector that the server uses. The garbage-collector metric names are unstable, and can change even in a minor JVM release.
Prometheus Metric Name Type Unit Description

idm_jvm_available_cpus

Gauge

Count

Number of processors available to the JVM. For more information, refer to Runtime.

idm_jvm_class_loading_loaded

Gauge

Count

Number of classes loaded since the Java virtual machine started. For more information, refer to ClassLoadingMXBean.

idm_jvm_class_loading_unloaded

Gauge

Count

Number of classes unloaded since the Java virtual machine started. For more information, refer to ClassLoadingMXBean.

idm_jvm_free_used_memory_bytes

Gauge

Bytes

For more information, refer to Runtime.

idm_jvm_garbage_collector_g1_old_generation_count

Gauge

Count

For each garbage collector in the JVM. For more information, refer to GarbageCollectorMXBean.

idm_jvm_garbage_collector_g1_old_generation_time

Gauge

Milliseconds

idm_jvm_garbage_collector_g1_young_generation_count

Gauge

Count

idm_jvm_garbage_collector_g1_young_generation_time

Gauge

Milliseconds

idm_jvm_max_memory_bytes

Gauge

Bytes

For more information, refer to Runtime.

idm_jvm_memory_usage_heap_committed

Gauge

Bytes

Amount of heap memory committed for the JVM to use. For more information, refer to MemoryMXBean.

idm_jvm_memory_usage_heap_init

Gauge

Bytes

idm_jvm_memory_usage_heap_max

Gauge

Bytes

Maximum amount of heap memory available to the JVM.

idm_jvm_memory_usage_heap_usage

Gauge

Bytes

idm_jvm_memory_usage_heap_used

Gauge

Bytes

Amount of heap memory used by the JVM.

idm_jvm_memory_usage_non_heap_committed

Gauge

Bytes

Amount of non-heap memory committed for the JVM to use.

idm_jvm_memory_usage_non_heap_init

Gauge

Bytes

Amount of non-heap memory the JVM initially requested from the operating system.

idm_jvm_memory_usage_non_heap_max

Gauge

Bytes

Maximum amount of non-heap memory available to the JVM.

idm_jvm_memory_usage_non_heap_usage

Gauge

Bytes

idm_jvm_memory_usage_non_heap_used

Gauge

Bytes

Amount of non-heap memory used by the JVM.

idm_jvm_memory_usage_pools_codeheap__non_nmethods__committed

Gauge

Bytes

For each pool. For more information, refer to MemoryPoolMXBean.

idm_jvm_memory_usage_pools_codeheap__non_nmethods__init

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_nmethods__max

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_nmethods__usage

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_nmethods__used

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_profiled_nmethods__committed

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_profiled_nmethods__init

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_profiled_nmethods__max

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_profiled_nmethods__usage

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__non_profiled_nmethods__used

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__profiled_nmethods__committed

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__profiled_nmethods__init

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__profiled_nmethods__max

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__profiled_nmethods__usage

Gauge

Bytes

idm_jvm_memory_usage_pools_codeheap__profiled_nmethods__used

Gauge

Bytes

idm_jvm_memory_usage_pools_compressed_class_space_committed

Gauge

Bytes

idm_jvm_memory_usage_pools_compressed_class_space_init

Gauge

Bytes

idm_jvm_memory_usage_pools_compressed_class_space_max

Gauge

Bytes

idm_jvm_memory_usage_pools_compressed_class_space_usage

Gauge

Bytes

idm_jvm_memory_usage_pools_compressed_class_space_used

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_eden_space_committed

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_eden_space_init

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_eden_space_max

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_eden_space_usage

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_eden_space_used

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_eden_space_used_after_gc

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_old_gen_committed

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_old_gen_init

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_old_gen_max

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_old_gen_usage

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_old_gen_used

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_old_gen_used_after_gc

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_survivor_space_committed

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_survivor_space_init

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_survivor_space_max

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_survivor_space_usage

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_survivor_space_used

Gauge

Bytes

idm_jvm_memory_usage_pools_g1_survivor_space_used_after_gc

Gauge

Bytes

idm_jvm_memory_usage_pools_metaspace_committed

Gauge

Bytes

idm_jvm_memory_usage_pools_metaspace_init

Gauge

Bytes

idm_jvm_memory_usage_pools_metaspace_max

Gauge

Bytes

idm_jvm_memory_usage_pools_metaspace_usage

Gauge

Bytes

idm_jvm_memory_usage_pools_metaspace_used

Gauge

Bytes

idm_jvm_memory_usage_total_committed

Gauge

Bytes

Amount of memory that is committed for the JVM to use. For more information, refer to MemoryMXBean.

idm_jvm_memory_usage_total_init

Gauge

Bytes

idm_jvm_memory_usage_total_max

Gauge

Bytes

idm_jvm_memory_usage_total_used

Gauge

Bytes

idm_jvm_thread_state_blocked_count

Gauge

Count

For more information, refer to ThreadMXBean.

idm_jvm_thread_state_count

Gauge

Count

Number of live threads including both daemon and non-daemon threads.

idm_jvm_thread_state_daemon_count

Gauge

Count

Number of live daemon threads.

idm_jvm_thread_state_new_count

Gauge

Count

Number of threads in the NEW state.

idm_jvm_thread_state_runnable_count

Gauge

Count

Number of threads in the RUNNABLE state.

idm_jvm_thread_state_terminated_count

Gauge

Count

Number of threads in the TERMINATED state.

idm_jvm_thread_state_timed_waiting_count

Gauge

Count

Number of threads in the TIMED_WAITING state.

idm_jvm_thread_state_waiting_count

Gauge

Count

Number of threads in the WAITING state.

idm_jvm_used_memory_bytes

Gauge

Bytes

For more information, refer to totalMemory().

Prometheus scheduler metrics available in IDM

Prometheus Metric Name Type Description

idm_scheduler.job{job-group=job-group, job-name=job-name, activity=completed}

Summary

A summary of completed jobs for the specified job-group and job-name.

idm_scheduler.job{job-group=job-group, job-name=job-name, activity=executed}

Timer

Time spent on executed jobs for the specified job-group and job-name.

idm_scheduler_job_store_repo_seconds{operation=operation, scheduler_object=scheduler-object}

Timer

Time spent storing scheduled jobs in the repository for the specified operation and scheduler-object.

idm_scheduler.trigger{activity=acquired, result=success}

Summary

A summary of successfully acquired jobs.

idm_scheduler.trigger{activity=acquired, result=timeout}

Summary

A summary of acquired jobs that time out.

idm_scheduler.trigger{activity=fired}

Summary

A summary of fired schedule triggers.

idm_scheduler.trigger{activity=misfired}

Summary

A summary of misfired schedule triggers.

idm_scheduler.trigger{activity=recovered}

Timer

Time spent on recovered triggers.

idm_scheduler_seconds{operation=operation, type=type}

Timer

Execution rate of scheduler requests for the specified type and operation.

Prometheus workflow metrics available in IDM

Prometheus Metric Name Type Description

idm_workflow_execution_action_seconds{action="message"}

Timer

Time spent invoking a message event.

idm_workflow_execution_action_seconds{action="signal"}

Timer

Time spent invoking a signal event.

idm_workflow_execution_action_seconds{action="trigger"}

Timer

Time spent triggering an execution.

idm_workflow_execution_query_seconds

Timer

Time spent querying executions.

idm_workflow_job_action_seconds{action="execute"}

Timer

Time spent forcing synchronous execution of a job.

idm_workflow_job_action_seconds{action="stacktrace"}

Timer

Time spent displaying the stacktrace for a job that triggered an exception.

idm_workflow_job_delete_seconds

Timer

Time spent deleting a job.

idm_workflow_job_query_seconds

Timer

Time spent querying jobs.

idm_workflow_job_read_seconds

Timer

Time spent reading a single job.

idm_workflow_jobdeadletter_action_seconds{action="execute"}

Timer

Time spent to execute dead-letter job.

idm_workflow_jobdeadletter_action_seconds{action="stacktrace"}

Timer

Time spent to retrieve the stacktrace for a dead-letter job.

idm_workflow_jobdeadletter_delete_seconds

Timer

Time spent to delete a dead letter job.

idm_workflow_jobdeadletter_query_seconds

Timer

Time spent to query dead letter jobs.

idm_workflow_jobdeadletter_read_seconds

Timer

Time spent to read a dead letter job.

idm_workflow_model_action_seconds{action="deploy"}

Timer

Time spent to deploy a model.

idm_workflow_model_action_seconds{action="list_deployments"}

Timer

Time spent to list model deployments.

idm_workflow_model_action_seconds{action="validate_bpmn"}

Timer

Time spent to validate BPMN content.

idm_workflow_model_create_seconds

Timer

Time spent to create a model.

idm_workflow_model_delete_seconds

Timer

Time spent to delete a model.

idm_workflow_model_query_seconds

Timer

Time spent to query models.

idm_workflow_model_read_seconds

Timer

Time spent to read a model.

idm_workflow_model_update_seconds

Timer

Time spent to update a model.

idm_workflow_processdefinition_delete_seconds

Timer

Time spent to delete a process definition.

idm_workflow_processdefinition_query_seconds

Timer

Time spent to query process definitions.

idm_workflow_processdefinition_read_seconds

Timer

Time spent to read a process definition.

idm_workflow_processinstance_action_seconds{action="migrate"}

Timer

Time spent to migrate a process instance.

idm_workflow_processinstance_action_seconds{action="validateMigration"}

Timer

Time spent to validate a migration of a process instance.

idm_workflow_processinstance_create_seconds

Timer

Time spent to create a process instance.

idm_workflow_processinstance_delete_seconds

Timer

Time spent to delete a process instance.

idm_workflow_processinstance_query_seconds

Timer

Time spent to query process instances.

idm_workflow_processinstance_read_seconds

Timer

Time spent to read a process instance.

idm_workflow_taskdefinition_query_seconds

Timer

Time spent to query task definitions.

idm_workflow_taskdefinition_read_seconds

Timer

Time spent to read a task definition.

idm_workflow_taskinstance_action_seconds{action="complete"}

Timer

Time spent to complete a task instance.

idm_workflow_taskinstance_query_seconds

Timer

Time spent to query task instances.

idm_workflow_taskinstance_read_seconds

Timer

Time spent to read a task instance.

idm_workflow_taskinstance_update_seconds

Timer

Time spent to update a task instance.