Web Agents 2024.9

Monitor services

The following sections describe how to set up and maintain monitoring in your deployment to ensure appropriate performance and service availability.

Monitor with Prometheus

Web Agent automatically exposes a monitoring endpoint where Prometheus can scrape metrics in a standard Prometheus format (version 0.0.4).

You can find information about installing and running Prometheus in the Prometheus documentation.

The Prometheus endpoint is protected by HTTP Basic Authentication. To access it, provide the agent URL, and the agent profile name and password. Always use HTTPS for secure connections to client applications.

The metrics returned are described in Metrics at the Prometheus endpoint.

Tools such as Grafana are available to create customized charts and graphs based on the information collected by Prometheus. Learn more on the Grafana website.

Access the Prometheus endpoint

  1. Install a Web Agent as described in the Installation, and use the agent to protect a web application. For example, set up the example in Policy enforcement.

  2. Access the Prometheus endpoint as follows, where https://agent.example.com:443 is the agent URL, web-agent is the agent profile name and password is the agent profile password:

    $ curl https://agent.example.com:443/agent/metrics --user web-agent:password

    The metrics are displayed:

    # TYPE policy_change counter
# HELP policy_change_total number of policy updates
policy_change_total{topic="notification"} 0
# TYPE config_change counter
# HELP config_change_total number of configuration changes
config_change_total{topic="notification"} 0
# TYPE not_enforced counter
# HELP not_enforced_total number of requests that were not enforced
not_enforced_total{topic="enforcement"} 0
...

Monitoring types

This section describes the data types used in monitoring:

Counter

Cumulative metric for a numerical value that only increases.

Gauge

Metric for a numerical value that can increase or decrease.

The value for a gauge is calculated when requested and represents the state of the metric at that specific time.

Histogram

Metric that samples observations, counts them in buckets, and provides a sum of all observed values.

Metrics at the Prometheus endpoint

Notification metrics

Web Agent exposes the following notification-related monitoring metrics:

Metric Type Description

policy_change_total

Counter

Number of policy change notifications received from Advanced Identity Cloud or AM.

config_change_total

Counter

Number of agent configuration change notifications received from Advanced Identity Cloud or AM.

Policy decision metrics

Web Agent exposes the following policy decision monitoring metrics:

Metric Type Description

not_enforced_total

Counter

Number of requests that weren’t enforced by the agent because of the not-enforced URL lists.

not_authorised_total

Counter

Number of requests denied by policy.

not_authenticated_total

Counter

Number of requests requiring authentication.

local_decision_total

Counter

Number of policy decisions the agent makes locally.

remote_decision_total

Counter

Number of policy decisions the agent requests from Advanced Identity Cloud or AM.

cache_decision_total

Counter

Number of policy decisions the agent takes from the cache.

Cache metrics

Web Agent exposes the following cache-related monitoring metrics:

Metric Type Description

cache_write_total

Counter

Number of session cache writes.

cache_update_total

Counter

Number of session cache updates.

cache_read_total

Counter

Number of session cache reads.

cache_miss_total

Counter

Number of sessions not found in cache.

cache_delete_total

Counter

Number of sessions deleted from cache.

cache_expiry_total

Counter

Number of sessions expired from cache.

cache_fault_total

Counter

Number of sessions that couldn’t be cached.

cache_occupancy

Gauge

Proportion of session cache that is occupied.

Connection metrics

Web Agent exposes the following connection-related monitoring metrics:

Metric Type Description

connection_total

Counter

Number of connections created.

connection_reuse_total

Counter

Number of cached connections reused.

Request metrics

Web Agent exposes the following request monitoring metrics:

Metric Type Description

policy_request_seconds

Histogram

Histogram of policy request times in seconds.

session_request_seconds

Histogram

Histogram of session request times in seconds.

config_request_seconds

Histogram

Histogram of configuration request times in seconds.

agent_time_seconds

Histogram

Histogram of agent time in request pipeline in seconds.

Monitor with the monitoring endpoint (deprecated)

The monitoring endpoint described in this section is deprecated. Use it only for diagnostics, in conjunction with Support.

A monitoring endpoint provides access to metrics for operations within the agent and between the agent an AM.

The monitoring endpoint is protected by HTTP Basic Authentication. To access it, provide the agent URL, and the agent profile name and password. Always use HTTPS for secure connections to client applications.

Metrics are displayed as a JSON response, with the fields described in Metrics at the monitoring endpoint (deprecated).

Access the monitoring endpoint

  1. Install a Web Agent as described in the Installation, and use the agent to protect a web application. For example, set up the example in Policy enforcement.

  2. Access the agent monitoring endpoint as follows, where https://agent.example.com:443 is the agent URL, and web-agent is the agent profile name.

    $ curl https://agent.example.com:443/agent/monitor --user web-agent

Enter host password for user 'web-agent':

  3. Enter the agent profile password to display the metrics:

    {
  "cache-invalidation": {
    "policy": 0,
    "profile": 1
  },
  "policy-decisions": {
    "neu": 0,
    "local": 0,
    "remote": 2,
    "cache": 0
    },
  "gc": {
    "runs": 1,
    "released": 0,
    "release-deferred": 0,
    "fill": 0.000000
  },
  "cache-operations": {
    "writes": 0,
    "rewrites": 2,
    "reads": 2,
    "misses": 0,
    "deletes": 0,
    "write-faults": 0,
    "expired": 0,
    "occupancy": 0
  },
  "connections": {
    "added": 2,
    "reused": 3
  }
}

Metrics at the monitoring endpoint (deprecated)

Metric Submetric Count of

cache-invalidation

policy

Number of policy change notifications received from AM.

profile

Number of agent configuration change notifications received from AM.

policy-decisions

neu

Number of requests that were not enforced by the agent because of the not-enforced URL lists.

local

Number of policy decisions the agent makes locally.

remote

Number of policy decisions the agent requests from AM.

cache

Number of policy decisions the agent takes from the cache.

gc

runs

Number of garbage collection runs.

released

Number of cache entries released during garbage collection runs.

release-defered

Number of entries with release deferred until the next garbage collection run.

fill

Floating point value between 0 and 1, representing the proportion of cache that is free after the most the recent garbage collection.

cache-operations

writes

Number of writes to cache.

rewrites

Number of updates to cache.

reads

Number of reads from cache.

misses

Number of failed searches of the cache.

deletes

Number of deletes from cache.

write-faults

Number of cache writes that fail because the cache is full.

expired

Number of expired cache entries.

occupancy

Proportion of cache that is occupied.

connections

added

Number of new connections made.

reused

Number of times existing connections were reused.