Monitoring server metrics with Prometheus

Prometheus is an open-source monitoring application that can be used to track numeric metrics over time. It uses HTTP GET requests to retrieve the metric data in the OpenMetrics text-based format.

The server includes an HTTP servlet extension that can publish the values of a configured set of numeric monitor attributes in this format so that they can be consumed by a Prometheus server.

Enabling Prometheus support in the server

The server is preconfigured with an HTTP servlet extension that can publish a wide variety of metrics for use by Prometheus.

To enable the servlet extension, add it to an HTTP or HTTPS connection handler, and either disable and re-enable the connection handler or restart the server. For example:

dsconfig set-connection-handler-prop \
  --handler-name "HTTPS Connection Handler" \
  --add "http-servlet-extension:Prometheus Monitoring" \
  --set enabled:false

dsconfig set-connection-handler-prop \
  --handler-name "HTTPS Connection Handler" \
  --set enabled:true

The servlet extension offers a number of configuration properties that can be used to customize its behavior. They include:

base-context-path

The path that the Prometheus server can use to consume any published metrics. By default, this is /metrics.

include-instance-name-label

Indicates whether all published metrics should include an instance label whose value is the name of the server instance.

include-product-name-label

Indicates whether all published metrics should include a product label whose value is the name of the server product, such as Ping Identity Authorize Server for PingAuthorize Server.

include-location-name-label

Indicates whether all published metrics should include a location label whose value is the name of the configured location for the server instance.

always-include-monitor-entry-name-label

Indicates whether all published metrics should include a monitor_entry label whose value is the name of the monitor entry (the value of the cn attribute) from which the metric was taken.

Even if this property is set to false, the server will still add this label to metrics in cases where there are multiple monitor entries with the same object class.

include-monitor-object-class-name-label

Indicates whether all published metrics should include a monitor_object_class label whose value is the name of the object class for the monitor entry from which the metric was taken.

include-monitor-attribute-name-label

Indicates whether all published metrics should include a monitor_attribute label whose value is the name of the monitor attribute from which the metric was taken.

label-name-value-pair

An optional set of name-value pairs for labels that should be included in all metrics published by the server. If provided, each item should use an equal sign to separate the label name from the value, such as label_name=label_value.

Label names must:

Start with a letter or underscore
Only contain letters, digits, and underscores
Be unique

Label values can be any non-empty string.

After the servlet extension has been enabled, you can verify that the metric information is available by retrieving it with a simple HTTP GET request using a web browser or a command-line tool like curl or wget.

For example, to retrieve the Prometheus metrics from the server paz.example.com using an HTTPS connection handler listening on port 8443 and the default base-context-path value of /metrics, use a URL of https://paz.example.com:8443/metrics.

Customizing published metrics

The server is preconfigured with a wide variety of definitions that publish the values of specified monitor attributes as metrics that can be consumed by Prometheus. However, you can also configure additional metrics by creating Prometheus monitor attribute metrics.

Each of these definitions includes the following configuration properties:

metric-name

The name of the metric as it should be published for use in Prometheus. The metric name must start with a letter or an underscore and must contain only letters, digits, and underscores. This is required, and all metric definitions must have a unique name.

monitor-attribute-name

The name of the monitor attribute whose value should be published as a Prometheus metric. This is required, and only single-valued attributes are supported. The value must be an integer or floating-point number.

monitor-object-class-name

The name of the object class for the monitor entry that contains the attribute to publish as a Prometheus metric. This is required.

metric-type

The data type for the metric to be published. This is required, and the value should be one of the following:

counter — Indicates that the value is one that increases over time as certain events occur in the server. This can include values that simply increment each time the event occurs, such as counting the number of connections accepted by the server since it started, but it can also include values that increase by larger amounts, such as counting the total number of bytes transferred in responses.
gauge — Indicates that the value is one that can go up or down over time and whose value represents the current state of some aspect of the server, such as the number of connections that are currently established.

filter

An optional search filter that can be used to indicate that the metric should only be published for monitor entries that match this filter. If this is not specified, then the metric is published for any monitor entry with the specified attribute and object class.

metric-description

An optional human-readable string that describes the purpose for the metric or that provides some other additional information about it.

label-name-value-pair

An optional set of name-value pairs for labels that should be included in the definition for the metric. If provided, each item should use an equal sign to separate the label name from the value, such as label_name=label_value. Label names must start with a letter or underscore and must only contain letters, digits, and underscores. Label values can be any non-empty string.

Metric-specific labels override server-wide labels, so if a metric-specific label is defined with the same name as a label that would otherwise be applied to all metrics, then the metric-specific value will be used instead of the server-wide value.

For example, to create a Prometheus metric that exposes the value of the currentConnections attribute of the general monitor entry, you could use a configuration change like the following:

dsconfig create-prometheus-monitor-attribute-metric \
  --extension-name "Prometheus Monitoring" \
  --metric-name general_monitor_current_connections \
  --set monitor-attribute-name:currentConnections \
  --set monitor-object-class-name:paz-general-monitor-entry \
  --set metric-type:gauge \
  --set "metric-description:The number of connections currently established"

If you want to remove any of the metrics that are included in the out-of-the-box configuration, then remove the definition for that metric from the server configuration.

Consuming metrics with Prometheus

After the server has been configured to publish its metrics, you can configure Prometheus to consume them.

The basic process is to update the server’s prometheus.yml file to add a job element to the scrape_configs section for the server from which the metrics should be retrieved, as in the following example:

- job_name: "paz.example.com:8443"
  metrics_path: "/metrics"
  scheme: "https"

  tls_config:
    ca_file: paz.example.com-ca-certificate.pem

  static_configs:
    - targets: ["paz.example.com:8443"]

See the Prometheus documentation for complete details.