--- title: Prometheus metrics description: IDM generates metrics only after a corresponding event occurs. For example, IDM doesn't create login-related metrics until a user logs in. If you're using a monitoring tool like Grafana, this may appear as missing data or empty panels on your dashboard. component: pingidm version: 8.1 page_id: pingidm:monitoring-guide:prometheus-metrics canonical_url: https://docs.pingidentity.com/pingidm/8.1/monitoring-guide/prometheus-metrics.html keywords: ["Monitoring"] section_ids: prometheus-metric-names: Prometheus general metrics available in IDM prometheus-jetty-metric-names: Prometheus Jetty metrics available in IDM prometheus-jvm-metric-names: Prometheus JVM metrics available in IDM prometheus-scheduler-metric-names: Prometheus scheduler metrics available in IDM prometheus-workflow-metric-names: Prometheus workflow metrics available in IDM --- # Prometheus metrics | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | IDM generates metrics only after a corresponding event occurs. For example, IDM doesn't create login-related metrics until a user logs in. If you're using a monitoring tool like Grafana, this may appear as missing data or empty panels on your dashboard. | Metrics accessed through the Prometheus endpoint are prepended with `idm_` and use underscores between words. For example, `idm_recon_target_phase_seconds`. In IDM, the available Prometheus metric types include: * [Summary](https://prometheus.io/docs/concepts/metric_types/#summary) * [Counter](https://prometheus.io/docs/concepts/metric_types/#counter) * [Gauge](https://prometheus.io/docs/concepts/metric_types/#gauge) Summary metrics include: * `metric_name_time-unit`: Contains `quantile` labels that provide the time-based distribution of measurements. * `metric_name_count`: Represents the number of times the metric has been timed. * `metric_name_time-unit_total`: Provides the total number of seconds measured by the metric per label group. For example: ```text idm_managed_seconds{managed_object="assignment",operation="queryCollection",quantile="0.5",} 9.728750000000001E-4 idm_managed_seconds{managed_object="assignment",operation="queryCollection",quantile="0.75",} 0.001247417 idm_managed_seconds{managed_object="assignment",operation="queryCollection",quantile="0.95",} 0.0016447500000000002 idm_managed_seconds{managed_object="assignment",operation="queryCollection",quantile="0.98",} 0.003216834 idm_managed_seconds{managed_object="assignment",operation="queryCollection",quantile="0.99",} 0.005315167 idm_managed_seconds{managed_object="assignment",operation="queryCollection",quantile="0.999",} 0.005315167 idm_managed_count{managed_object="assignment",operation="queryCollection",} 88.0 idm_managed_seconds_total{managed_object="assignment",operation="queryCollection",} 0.098369748 idm_managed_seconds{managed_object="organization",operation="create",quantile="0.5",} 0.007266166 idm_managed_seconds{managed_object="organization",operation="create",quantile="0.75",} 0.008662667 idm_managed_seconds{managed_object="organization",operation="create",quantile="0.95",} 0.028109959 idm_managed_seconds{managed_object="organization",operation="create",quantile="0.98",} 0.08822537500000001 idm_managed_seconds{managed_object="organization",operation="create",quantile="0.99",} 0.08822537500000001 idm_managed_seconds{managed_object="organization",operation="create",quantile="0.999",} 0.08822537500000001 idm_managed_count{managed_object="organization",operation="create",} 31.0 idm_managed_seconds_total{managed_object="organization",operation="create",} 0.373556921 ``` | | | | - | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | For Summary metrics that don't include the `quantile` grouping (designated with a footnote in the following table), there's a known issue where the `_total` metric isn't a total of the entire series. Instead, the `_total` metric is a total of each label group, which leads to identical metrics for the `_total` and `count`.For example, the audit summary metric produces the following output:```text idm_audit_total{audit_topic="access",} 571.0 idm_audit_count{audit_topic="access",} 571.0 idm_audit_total{audit_topic="activity",} 470.0 idm_audit_count{audit_topic="activity",} 470.0 idm_audit_total{audit_topic="authentication",} 548.0 idm_audit_count{audit_topic="authentication",} 548.0 ``` | | | | | - | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | Deprecated metrics are still available until they're removed in a future release. Learn more in [Deprecated metric collection](monitoring.html#deprecated-metric-collection). | ## Prometheus general metrics available in IDM | Metric Name | Label / Type | Description | | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `idm_audit`\[[1](#_footnotedef_1 "View footnote.")] | `{audit_topic=audit-topic}` Summary | Count of all audit events generated of a given topic type. | | `idm_custom_endpoint_seconds` | `{name=endpoint-name,request_type=request-type}` Summary | Rate of calls to a custom endpoint script and the time taken to perform this operation. | | `idm_field_augmentation` | `{origin-type=edge\|vertex}` Summary | Rate of reading response objects to fulfill the `_fields` requested (when the fields weren't populated by the initial repo query). | | []()`idm_router_filter_seconds`\[[2](#_footnotedef_2 "View footnote.")]\[[3](#_footnotedef_3 "View footnote.")] | `{action=action,name=name,script_name=script_name,quantile=quantile,system=system}` Summary | Rate that filter scripts are executed, per action. Monitors scripted filters and delegated administration. | | `idm_icf_connector_server_availability` | `{name=system-identifier,type=connector-server-type}` Gauge | Status of the connector server. A value of `1` indicates the server is running. A value of `0` indicates the server isn't running. | | []()`idm_icf_pending`\[[4](#_footnotedef_4 "View footnote.")] | `{action="authentication",bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="action",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `authentication` actions over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="create",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `create` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="delete",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `delete` operations over the configured limit. | | | `{action="liveSync",bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="action",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `liveSync` actions on a system object. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="patch",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `patch` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="read",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `read` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="update",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `update` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="query",query_type="_queryExpression",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `queryExpression` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="query",query_type="_queryFilter",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `queryFilter` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="query",query_type="_queryId",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `queryId` operations over the configured limit. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,operation="query",query_type="_UNKNOWN",system_identifier=system_identifier,quantile=quantile}` Gauge | The number of pending `UNKNOWN` operations over the configured limit. | | `idm_icf_seconds` | `{action="authenticate",bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="action",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF `authentication` actions and the action performance time for the given connector. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="create",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF `create` operations and the operation performance time for the given connector. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="delete",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF `delete` operations and the operation performance time for the given connector. | | | []()`{action="liveSync",bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="action",system_identifier=system_identifier,quantile=quantile}` Summary | Duration of live sync on a system object. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="patch",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF `patch` operations and the operation performance time for the given connector. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="read",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF `read` operations and the operation performance time for the given connector. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="update",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF `update` operations and the operation performance time for the given connector. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="query",query_type=queryExpression,system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF query executions with `queryExpression` and the time taken to perform this operation. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="query",query_type=queryFilter,system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF query executions with `queryFilter` and the time taken to perform this operation. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="query",query_type=queryId,system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF query executions with `queryId` and the time taken to perform this operation. | | | `{bundle_version=bundle_version,connector=connector,connector_type=connector_type,location=location,object_class=objectClass,operation="query",query_type="UNKNOWN",system_identifier=system_identifier,quantile=quantile}` Summary | Rate of ICF query executions when the query type is `UNKNOWN` and the time taken to perform this operation. | | `idm_internal_relationship_fetch_relationship_fields_seconds` | `{internal_object=internal-object}` Summary | Rate of fetch operations of relationship fields for internal objects. | | `idm_internal_role_relationship_get_relationship_value_for_resource_seconds` | Summary | Query rate on relationship values for internal objects. | | `idm_internal_relationship_validate_relationship_fields_seconds` | `{internal-object=internal-object}` Summary | Rate of validate operations of relationship fields for internal objects. | | `idm_internal_seconds` | `{internal_object=internal-object,operation=operation}` Summary | Rate of operations on internal objects. | | `idm_live_sync_seconds` | `{object-type=object-type,system-name=system-name}` Summary | Duration of live sync on a system object. | | `idm_managed_relationship_fetch_relationship_fields_seconds` | Summary | Rate of fetches of relationship fields of a managed object. | | `idm_managed_managed-object_relationship_get_relationship_value_for_resource_seconds` | Summary | Rate of queries to get relationship values for a resource on a managed object. | | `idm_managed_relationship_validate_relationship_fields_seconds` | Summary | Rate of validations of relationship fields of a managed object. | | `idm_managed_seconds` | `{managed_object=managed-object,operation=operation}` Summary | Rate of operations on a managed object. | | []()`idm_managed_script_hook_seconds`\[[5](#_footnotedef_5 "View footnote.")]\[[3](#_footnotedef_3 "View footnote.")] | `{object=object, script_hook=script_hook}` Summary | Rate of executions of a script on a managed object. | | `idm_managed_object_handle_temporal_constraints_on_create` | Summary | Latency of enforcing temporal constraints on role objects during object creation. | | `idm_managed_object_handle_temporal_constraints_on_delete` | Summary | Latency of enforcing temporal constraints on role objects during object deletion. | | `idm_managed_object_handle_temporal_constraints_on_update` | Summary | Latency of enforcing temporal constraints on role objects during object update. | | `idm_managed_relationship_handle_temporal_constraints_on_create` | Summary | Latency of enforcing temporal constraints on relationship grants during edge creation. | | `idm_managed_relationship_handle_temporal_constraints_on_delete` | Summary | Latency of enforcing temporal constraints on relationship grants during edge deletion. | | `idm_managed_relationship_handle_temporal_constraints_on_update` | Summary | Latency of enforcing temporal constraints on relationship grants during edge update. | | `idm_managed_relationship_validate_read_relationship_endpoint_edges_seconds` | Summary | Rate of reads on relationship endpoint edges for validation. | | `idm_null_array_filter_augmentation_seconds` | `{request_type=requestType}` Summary | Time spent in filter that maps non-nullable, null-valued array fields to an empty array. This filter is traversed for all repository access relating to internal and managed objects. | | `idm_recon_assoc_entry_merged_query_merge_results_seconds` | Summary | Rate of merge operations after source or target objects are retrieved during a merged query of reconciled association entries. | | `idm_recon_assoc_entry_merged_query_page_assoc_entries_seconds` | Summary | Rate of individual paged reconciled association entry queries during a merged query. More than one page of entries might be requested to build a single page of merged results. | | `idm_recon_assoc_entry_merged_query_query_source_seconds` | Summary | Rate of source object retrieval using a query when merging source objects to reconciled association entries. | | `idm_recon_assoc_entry_merged_query_query_target_seconds` | Summary | Rate of target object retrieval using a query when merging target objects to reconciled association entries. | | `idm_recon_association-persistence` | `{recon-id=reconId,operation=operation}` Summary | The time taken to persist association data. The operation can be `source`, `target`, or `amendsource`, depending on whether data is produced for a source-phase or target-phase reconciliation association, or to amend the association for a specific source. | | `idm_recon_id_queries_phase_seconds` | Summary | Rate of executions of the ID query phase of a reconciliation and the time taken to perform this operation. | | `idm_recon_seconds` | Summary | Rate of executions of a full reconciliation and the time taken to perform this operation. | | `idm_recon_source_phase_page_seconds` | Summary | Rate of pagination executions of the source phase of a reconciliation and the time taken to perform this operation. | | `idm_recon_source_phase_seconds` | Summary | Rate of executions of the source phase of a reconciliation and the time taken to perform this operation. | | `idm_recon_target_phase_seconds` | Summary | Rate of executions of the target phase of a reconciliation and the time taken to perform this operation. | | `idm_repo_seconds` | `{action_name=action-name,command=command,operation=operation,repo_type=repo-type,resource_mapping=resource-mapping}` Summary | Rate of actions to a repository datasource for a generic or explicit mapped table. | | | `{operation=operation,repo_type=repo-type,resource_mapping=resource_mapping,query_type=query_type,action_name=action_name,command=command}` Summary | Rate of filtered queries (using [native query expressions](../objects-guide/queries.html#native-queries)) on the relationship table. This metric measures the time spent running the query and the number of times it's invoked. | | | `{operation=operation,repo_type=repo-type,resource_mapping=resource_mapping,query_type=queryFilter,action_name=action_name,command=command}` Summary | Rate of filtered queries using the `_queryFilter` parameter on the relationship table. This metric measures the time spent running the query and the number of times it's invoked. | | `idm_repo_execute_seconds` | `{operation=create_properties,repo_type=repo-type,resource_mapping=resource-mapping}` Summary | Rate of execution time on the JDBC database for the `create_properties` operations. This operation is performed for every generic object `create` when it persists the searchable properties. The rate measured here doesn't include the time taken to connect to the database from the connection pool. The physical connections to the database have already been established inside the connection pool. | | | `{operation=operation,repo_type=repo-type,resource_mapping=resource-mapping}` Summary | Rate of execution time on the JDBC database for CRUD operations. This rate doesn't include the time taken to connect to the database from the connection pool. The physical connections to the database have already been established inside the connection pool. | | | `{operation="query",queryType=queryFilter\|queryId,repo_type=repo-type,resource_mapping=resource-mapping}` Summary | Rate of execution time on the JDBC database for queries (either `queryFilter` or `queryId`). This rate doesn't include the time taken to connect to the database from the connection pool. The physical connections to the database have already been established inside the connection pool. | | `idm_repo_get_connection_seconds` | `{repo_type=repo-type}` Summary | Rate of retrievals of a repository connection. | | `idm_repo_jdbc_cache_objecttypes_count` | `{event="hit\|miss",type=resource-mapping}` Counter | Counts the usage statistics of the `objecttypeid` cache that maps an object type to its `objecttypeid`. The expected count is a small number of misses (sometimes, only one) and the remainder of hits. | | `idm_repo_jdbc_relationship_edge_execute_seconds` | `{joinedToVertex=joinedToVertex>}` Summary | Time spent running the Edge→Vertex relationship join query on the database and collecting the result set. | | `idm_repo_jdbc_relationship_edge_process_seconds`\[[1](#_footnotedef_1 "View footnote.")] | `{joinedToVertex=joinedToVertex}` Summary | The amount of time taken to process the results returned from the query measured by `idm_repo_jdbc_relationship_edge_execute_seconds`, but not the query itself. | | `idm_repo_jdbc_relationship_find_referenced_collections_seconds` \[[1](#_footnotedef_1 "View footnote.")] | Summary | Measures the SQL execution time to find referenced collections on the Edge→Vertex relationship. | | `idm_repo_jdbc_relationship_vertex_execute_seconds` | Summary | Rate of relationship graph query execution times. | | `idm_repo_jdbc_relationship_vertex_process_seconds` | Summary | Rate of relationship graph query result processing times. | | `idm_repo_raw__queryid_credential_queryId_seconds` | Summary | Execution rate of a query with `queryId` at a repository level and the time taken to perform this operation. | | `idm_repo_relationship_count` | `{operation=operation,origin_type=origin_type,repo_type=repo_type,stage=stage}` Counter | The count of all repository relationship calls referenced by queried objects. | | `idm_repo_relationship_seconds` | `{operation=operation,repo_type=repo-type}` Summary | Time of CRUDPAQ operations to a repository datasource for a generic, explicit, or relationship mapped table. | | | `{operation=operation,repo_type=repo-type,resource_mapping=resource-mapping}` Summary | Rate of initiations of a CRUDPAQ operation to a repository datasource. | | `idm_router_path-name_action_action-type_seconds` | Summary | Rate of actions over the router and the time taken to perform this operation. | | `idm_router_path-name_create_seconds` | Summary | Rate of creates over the router and the time taken to perform this operation. | | `idm_router_path-name_delete_seconds` | Summary | Rate of deletes over the router and the time taken to perform this operation. | | `idm_router_path-name_patch_seconds` | Summary | Rate of patches over the router and the time taken to perform this operation. | | `idm_router_path-name_query_queryExpression_seconds` | Summary | Rate of queries with `queryExpression` completed over the router and the time taken to perform this operation. | | `idm_router_path-name_query_queryFilter_seconds` | Summary | Rate of queries with `queryFilter` completed over the router and the time taken to perform this operation. | | `idm_router_path-name_read_seconds` | Summary | Rate of reads over the router and the time taken to perform this operation. | | `idm_router_path-name_update_seconds` | Summary | Rate of updates over the router and the time taken to perform this operation. | | `idm_sync_create_object_seconds` | Summary | Rate of requests to create an object on the target and the time taken to perform this operation. | | `idm_sync_delete_target_seconds` | Summary | Rate of requests to delete an object on the target and the time taken to perform this operation. | | `idm_sync_objectmapping_seconds` | `{mapping_name=mapping-name}` Summary | Rate of configurations applied to a mapping. | | `idm_sync_queue_acquire` | `{mapping_name=mapping-name, action=action}` Summary | Rate of acquisition of queued synchronization events from the queue. | | `idm_sync_queue_discard` | `{mapping_name=mapping-name, action=action}` Summary | Rate of deletion of synchronization events from the queue. | | `idm_sync_queue_execution` | `{mapping_name=mapping-name, action=action}` Summary | Rate at which queued synchronization operations are run. | | `idm_sync_queue_failed` \[[1](#_footnotedef_1 "View footnote.")] | `{mapping_name=mapping-name, action=action}` Summary | Number of queued synchronization operations that failed. | | `idm_sync_queue_poll_pending_events` | `{mapping_name=mapping-name}` Summary | The polling latency for synchronization events. | | `idm_sync_queue_precondition_failed` \[[1](#_footnotedef_1 "View footnote.")] | `{mapping_name=mapping-name, action=action}` Summary | Number of queued synchronization events acquired by another node in the cluster. | | `idm_sync_queue_query_previously_acquired_events_seconds` \[[1](#_footnotedef_1 "View footnote.")] | `{mapping_name=mapping-name}` Summary | Measures the amount of time spent querying for synchronization events previously acquired by another node that have now been acquired by this node due to a mapping rebalancing. | | `idm_sync_queue_rejected_executions` \[[1](#_footnotedef_1 "View footnote.")] | `{mapping_name=mapping-name, action=action}` Summary | Number of queued synchronization events rejected because the backing thread-pool queue reached full capacity and the thread pool had allocated its maximum-configured number of threads. | | `idm_sync_queue_release` | `{mapping_name=mapping-name, action=action}` Summary | Rate at which queued synchronization events are released. | | `idm_sync_queue_submit` | `{mapping_name=mapping-name, action=action}` Summary | Insertion rate of synchronization events into the queue. | | `idm_sync_raw_read_object_seconds` | Summary | Rate of reads of an object. | | `idm_sync_source_assess_situation_seconds` | Summary | Assessment rate of a synchronization situation. | | `idm_sync_source_correlate_target_seconds` | Summary | Correlation rate between a target and a given source, and the time taken to perform this operation. | | `idm_sync_source_determine_action_seconds` | Summary | Determination rate on a synchronization action based on its current situation. | | `idm_sync_source_perform_action_seconds` | Summary | Completion rate of an action performed on a synchronization operation. | | `idm_sync_target_assess_situation_seconds` | Summary | Assessment rate of a target situation. | | `idm_sync_target_determine_action_seconds` | Summary | Determination rate on a target action based on its current situation. | | `idm_sync_target_perform_action_seconds` | Summary | Completion rate of an action performed on a target synchronization operation. | | `idm_sync_update_target_seconds` | Summary | Request rate to update an object on the target and the time taken to perform this operation. | | `idm_user_login` \[[1](#_footnotedef_1 "View footnote.")] | `{user_type=user-type}` Summary | Count of all successful logins by user type. | | `idm_user_login_total` \[[1](#_footnotedef_1 "View footnote.")] | `{provider=provider,user_type=user-type}` Summary | Count of all successful logins by user type and provider. | | `idm_virtualpropertiesfromrelationships_notfound` \[[1](#_footnotedef_1 "View footnote.")] | `{virtual_properties=virtual_properties[6],relationship_traversal=relationship_resource_path}[7]` Summary | Number of 404 responses encountered when querying the `resource_collection`/`relationship_field` specified in the `traversal_depthX` tag for the most recent X. X corresponds to the relationship field sequence. | | `idm_virtualpropertiesfromrelationships_unsatisfiedtempconstraint`\[[1](#_footnotedef_1 "View footnote.")] | `{virtual_properties=virtual_properties[6],relationship_traversal=relationship_resource_path}[7]` Summary | Number of edges skipped due to an unsatisfied temporal constraint on either the edge or the referred-to vertex. Encountered when querying the resource collection and relationship field at the `traversal_depthX` tag for the most recent X. X corresponds to the relationship field sequence. | | `idm_virtualpropertiesfromrelationships_seconds`\[[1](#_footnotedef_1 "View footnote.")] | `{virtual_properties=virtual_properties[6],relationship_resource_path=relationship_resource_path}[7]` Summary | Time spent traversing relationship fields to calculate the specified virtual properties. The managed objects linked to by the traversal relationship fields define a tree, whose root is the virtual property host. This object tree is traversed depth-first, with the `traversal_depthX` corresponding to the latency involved with each relationship traversal. `Traversal_depth0` corresponds to the first relationship field traversed. Because the tree is traversed depth-first, `traversal_depthX` will subsume all the traversal latencies for all `traversal_depthY`, where Y>X. X corresponds to the relationship field sequence. | | `idm_edge_to_vertex_relationship_created_seconds` \[[1](#_footnotedef_1 "View footnote.")] | Summary | Time it takes to create the Edge→Vertex relationship. | | `idm_edge_to_vertex_relationship_notification_seconds` \[[1](#_footnotedef_1 "View footnote.")] | Summary | Time it takes to provide notifications on the Edge→Vertex relationship. | ## Prometheus Jetty metrics available in IDM These metrics include Jetty thread pool and request metrics. | Prometheus Metric Name | Type | Unit | Description | | ----------------------------------------------------------------------- | ------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | []()`idm_jetty_qos_queue_count`\[[8](#_footnotedef_8 "View footnote.")] | Gauge | Count | Current number of requests queued in the [Jetty QoSHandler](../install-guide/idm-config-properties-jetty.html#config-jetty-qos-handler) queue. | | []()`idm_jetty_qos_queue_count_max` | Gauge | Count | Maximum number of requests that can be queued. | | []()`idm_jetty_qos_concurrent_max` | Gauge | Count | Maximum number of requests that can be handled concurrently. | | []()`idm_jetty_qos_queue_milliseconds_max` | Gauge | Count | Maximum amount of time a request can be queued. | | `idm_jetty_thread_queue`\[[8](#_footnotedef_8 "View footnote.")] | Gauge | Count | Size of the job queue. | | `idm_jetty_thread{thread_type="ready"}` | Gauge | Count | Number of threads ready to run transient jobs, such as handling requests. | | `idm_jetty_thread{thread_type="leased"}` | Gauge | Count | Number of threads used by internal Jetty components. | | `idm_jetty_thread{thread_type="reserved"}` | Gauge | Count | Number of available threads reserved for queue management. | | `idm_jetty_thread{thread_type="idle"}` | Gauge | Count | Number of idle threads that aren't reserved. | | `idm_jetty_thread{thread_type="utilized"}` | Gauge | Count | Number of threads currently running transient jobs, such as handling requests. | | `idm_jetty_thread_total` | Gauge | Count | Total number of threads in the pool. | | `idm_jetty_thread_islowonthreads` | Gauge | Count | Whether the pool is low on threads. `1` if true, `0` otherwise. | | `idm_jetty_request_active` | Gauge | Count | Current number of active requests. | | `idm_jetty_request_max` | Counter | Count | Maximum number of concurrently active requests. | | `idm_jetty_request_failed{error_code="4xx"}` | Counter | Count | Number of requests with a `4xx` response status. | | `idm_jetty_request_failed{error_code="5xx"}` | Counter | Count | Number of requests with a `5xx` response status. | | `idm_jetty_request_nanoseconds_max` | Gauge | Nanoseconds | Maximum request run time. | | `idm_jetty_request_nanoseconds_stddev` | Gauge | Nanoseconds | Standard deviation for request run time. | | `idm_jetty_request_servlet_active` | Gauge | Count | Current number of requests the servlets handle. | | `idm_jetty_request_servlet_max` | Counter | Count | Maximum number of requests the servlets handle concurrently. | | `idm_jetty_request_servlet_nanoseconds_max` | Gauge | Nanoseconds | Maximum servlet run time. | | `idm_jetty_request_servlet_nanoseconds_stddev` | Gauge | Nanoseconds | Standard deviation for servlet run time. | ## Prometheus JVM metrics available in IDM | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | | These metrics depend on the JVM version and configuration. In particular, garbage-collector-related metrics depend on the garbage collector that the server uses. The garbage-collector metric names are unstable and can change even in a minor JVM release. | | Prometheus Metric Name | Type | Unit | Description | | ------------------------------------------------------------------------------- | ------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `idm_jvm_available_cpus` | Gauge | Count | Number of processors available to the JVM. Learn more in [Runtime](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Runtime.html). | | `idm_jvm_class_loading_loaded_total` | Counter | Count | Number of classes loaded since the Java virtual machine started. Learn more in [ClassLoadingMXBean](https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/ClassLoadingMXBean.html). | | `idm_jvm_class_loading_unloaded_total` | Counter | Count | Number of classes unloaded since the Java virtual machine started. Learn more in [ClassLoadingMXBean](https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/ClassLoadingMXBean.html). | | `idm_jvm_free_memory_bytes` | Gauge | Bytes | Learn more in [Runtime](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Runtime.html). | | `idm_jvm_garbage_collector_count_total{name="G1-Old-Generation"}` | Counter | Count | For each garbage collector in the JVM. Learn more in [GarbageCollectorMXBean](https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/GarbageCollectorMXBean.html). | | `idm_jvm_garbage_collector_time_total{name="G1-Old-Generation"}` | Counter | Milliseconds | | | `idm_jvm_garbage_collector_count_total{name="G1-Young-Generation"}` | Counter | Count | | | `idm_jvm_garbage_collector_time_total{name="G1-Young-Generation"}` | Counter | Milliseconds | | | `idm_jvm_max_memory_bytes` | Gauge | Bytes | Learn more in [Runtime](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Runtime.html). | | `idm_jvm_memory_usage_committed{location="heap"}` | Gauge | Bytes | Amount of heap memory committed for the JVM to use. Learn more in [MemoryMXBean](https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/MemoryMXBean.html). | | `idm_jvm_memory_usage_init{location="heap"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_max{location="heap"}` | Gauge | Bytes | Maximum amount of heap memory available to the JVM. | | `idm_jvm_memory_usage_used{location="heap"}` | Gauge | Bytes | Amount of heap memory used by the JVM. | | `idm_jvm_memory_usage_committed{location="non-heap"}` | Gauge | Bytes | Amount of non-heap memory committed for the JVM to use. | | `idm_jvm_memory_usage_init{location="non-heap"}` | Gauge | Bytes | Amount of non-heap memory the JVM initially requested from the operating system. | | `idm_jvm_memory_usage_max{location="non-heap"}` | Gauge | Bytes | Maximum amount of non-heap memory available to the JVM. | | `idm_jvm_memory_usage_used{location="non-heap"}` | Gauge | Bytes | Amount of non-heap memory used by the JVM. | | `idm_jvm_memory_usage_pools_committed{name="CodeHeap-'non-nmethods'"}` | Gauge | Bytes | For each pool. Learn more in [MemoryPoolMXBean](https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/MemoryPoolMXBean.html). | | `idm_jvm_memory_usage_pools_init{name="CodeHeap-'non-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="CodeHeap-'non-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="CodeHeap-'non-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="CodeHeap-'non-profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="CodeHeap-'non-profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="CodeHeap-'non-profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="CodeHeap-'non-profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="CodeHeap-'profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="CodeHeap-'profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="CodeHeap-'profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="CodeHeap-'profiled-nmethods'"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="Compressed-Class-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="Compressed-Class-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="Compressed-Class-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="Compressed-Class-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="G1-Eden-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="G1-Eden-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="G1-Eden-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="G1-Eden-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used_after_gc{name="G1-Eden-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="G1-Old-Gen"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="G1-Old-Gen"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="G1-Old-Gen"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="G1-Old-Gen"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used_after_gc{name="G1-Old-Gen"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="G1-Survivor-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="G1-Survivor-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="G1-Survivor-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="G1-Survivor-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used_after_gc{name="G1-Survivor-Space"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_committed{name="Metaspace"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_init{name="Metaspace"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_max{name="Metaspace"}` | Gauge | Bytes | | | `idm_jvm_memory_usage_pools_used{name="Metaspace"}` | Gauge | Bytes | | | `idm_jvm_thread_state{state="blocked"}` | Gauge | Count | Learn more in [ThreadMXBean](https://docs.oracle.com/en/java/javase/21/docs/api/java.management/java/lang/management/ThreadMXBean.html). | | `idm_jvm_thread_state_daemon` | Gauge | Count | Number of live daemon threads. | | `idm_jvm_thread_state{state="new"}` | Gauge | Count | Number of threads in the `NEW` state. | | `idm_jvm_thread_state{state="runnable"}` | Gauge | Count | Number of threads in the `RUNNABLE` state. | | `idm_jvm_thread_state{state="terminated"}` | Gauge | Count | Number of threads in the `TERMINATED` state. | | `idm_jvm_thread_state{state="timed_waiting"}` | Gauge | Count | Number of threads in the `TIMED_WAITING` state. | | `idm_jvm_thread_state{state="waiting"}` | Gauge | Count | Number of threads in the `WAITING` state. | | `idm_jvm_used_memory_bytes` | Gauge | Bytes | Learn more in [totalMemory()](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Runtime.html#totalMemory\(\)). | | | | | - | ------------------------------------------------------------------------------------------------------------------------------------------- | | | [Deprecated](../release-notes/deprecated-functionality.html#deprecation-jvm-mem-usage-metrics) metrics are not shown in the previous table. | ## Prometheus scheduler metrics available in IDM | Prometheus Metric Name | Type | Description | | --------------------------------------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------- | | `idm_scheduler_job{job-group=job-group,job-name=job-name,activity=completed}` | Summary | A summary of completed jobs for the specified job-group and job-name. | | `idm_scheduler_job{job-group=job-group,job-name=job-name,activity=executed}` | Summary | Time spent on executed jobs for the specified job-group and job-name. | | `idm_scheduler_job_store_repo_seconds{operation=operation,scheduler_object=scheduler-object}` | Summary | Time spent storing scheduled jobs in the repository for the specified operation and scheduler-object. | | `idm_scheduler_trigger{activity=acquired,result=success}` | Summary | A summary of successfully acquired jobs. | | `idm_scheduler_trigger{activity=acquired,result=timeout}` | Summary | A summary of acquired jobs that time out. | | `idm_scheduler_trigger{activity=fired}` | Summary | A summary of fired schedule triggers. | | `idm_scheduler_trigger{activity=misfired}` | Summary | A summary of misfired schedule triggers. | | `idm_scheduler_trigger{activity=recovered}` | Summary | Time spent on recovered triggers. | | `idm_scheduler_seconds{operation=operation,type=type}` | Summary | Execution rate of scheduler requests for the specified type and operation. | ## Prometheus workflow metrics available in IDM | Prometheus Metric Name | Type | Description | | ------------------------------------------------------------------------- | ------- | --------------------------------------------------------------------------- | | `idm_workflow_execution_action_seconds{action="message"}` | Summary | Time spent invoking a message event. | | `idm_workflow_execution_action_seconds{action="signal"}` | Summary | Time spent invoking a signal event. | | `idm_workflow_execution_action_seconds{action="trigger"}` | Summary | Time spent triggering an execution. | | `idm_workflow_execution_query_seconds` | Summary | Time spent querying executions. | | `idm_workflow_job_action_seconds{action="execute"}` | Summary | Time spent forcing synchronous execution of a job. | | `idm_workflow_job_action_seconds{action="stacktrace"}` | Summary | Time spent displaying the stacktrace for a job that triggered an exception. | | `idm_workflow_job_delete_seconds` | Summary | Time spent deleting a job. | | `idm_workflow_job_query_seconds` | Summary | Time spent querying jobs. | | `idm_workflow_job_read_seconds` | Summary | Time spent reading a single job. | | `idm_workflow_jobdeadletter_action_seconds{action="execute"}` | Summary | Time spent to execute dead-letter job. | | `idm_workflow_jobdeadletter_action_seconds{action="stacktrace"}` | Summary | Time spent to retrieve the stacktrace for a dead-letter job. | | `idm_workflow_jobdeadletter_delete_seconds` | Summary | Time spent to delete a dead letter job. | | `idm_workflow_jobdeadletter_query_seconds` | Summary | Time spent to query dead letter jobs. | | `idm_workflow_jobdeadletter_read_seconds` | Summary | Time spent to read a dead letter job. | | `idm_workflow_model_action_seconds{action="deploy"}` | Summary | Time spent to deploy a model. | | `idm_workflow_model_action_seconds{action="list_deployments"}` | Summary | Time spent to list model deployments. | | `idm_workflow_model_action_seconds{action="validate_bpmn"}` | Summary | Time spent to validate BPMN content. | | `idm_workflow_model_create_seconds` | Summary | Time spent to create a model. | | `idm_workflow_model_delete_seconds` | Summary | Time spent to delete a model. | | `idm_workflow_model_query_seconds` | Summary | Time spent to query models. | | `idm_workflow_model_read_seconds` | Summary | Time spent to read a model. | | `idm_workflow_model_update_seconds` | Summary | Time spent to update a model. | | `idm_workflow_processdefinition_delete_seconds` | Summary | Time spent to delete a process definition. | | `idm_workflow_processdefinition_query_seconds` | Summary | Time spent to query process definitions. | | `idm_workflow_processdefinition_read_seconds` | Summary | Time spent to read a process definition. | | `idm_workflow_processinstance_action_seconds{action="migrate"}` | Summary | Time spent to migrate a process instance. | | `idm_workflow_processinstance_action_seconds{action="validateMigration"}` | Summary | Time spent to validate a migration of a process instance. | | `idm_workflow_processinstance_create_seconds` | Summary | Time spent to create a process instance. | | `idm_workflow_processinstance_delete_seconds` | Summary | Time spent to delete a process instance. | | `idm_workflow_processinstance_query_seconds` | Summary | Time spent to query process instances. | | `idm_workflow_processinstance_read_seconds` | Summary | Time spent to read a process instance. | | `idm_workflow_taskdefinition_query_seconds` | Summary | Time spent to query task definitions. | | `idm_workflow_taskdefinition_read_seconds` | Summary | Time spent to read a task definition. | | `idm_workflow_taskinstance_action_seconds{action="complete"}` | Summary | Time spent to complete a task instance. | | `idm_workflow_taskinstance_query_seconds` | Summary | Time spent to query task instances. | | `idm_workflow_taskinstance_read_seconds` | Summary | Time spent to read a task instance. | | `idm_workflow_taskinstance_update_seconds` | Summary | Time spent to update a task instance. | *** [1](#_footnoteref_1). This summary metric doesn't include the quantile grouping and only has the metric\_name\_count and metric\_name\_total entries.[2](#_footnoteref_2). The idm\_router\_filter\_seconds metric name replaces the "filter" name. This metric can specify a "name" and "system" label. If no "name" label is specified, it defaults to "unknown". The "system" label is always system="false".[3](#_footnoteref_3). The deprecated metric names are still available and are generated along with the new metric names unless "deprecatedMetricsEnabled" is set to "false" in "conf/metrics.json".[4](#_footnoteref_4). A pending request gauge won't register until the associated RequestType has been invoked at least one time.[5](#_footnoteref_5). This metric naming convention replaces idm\_managed\_seconds{managed\_object="managed\_object",operation="operation\_name",script="script\_name"} and includes optional "object" and "script\_hook" labels.[6](#_footnoteref_6). Each virtual property is listed in both the label name and value. For example: effectiveAssignments\_effectiveRoles="effectiveAssignments,effectiveRoles"[7](#_footnoteref_7). The relationship resource path is used as both the label name and value. For example: managed\_user\_roles="managed\_user.roles"[8](#_footnoteref_8). The QoSHandler metric, idm\_jetty\_qos\_queue\_count, accurately contains the number of queued requests in the handler queue and replaces the idm\_jetty\_thread\_queue metric.