Web Agents 2024.9

Maintenance guide

This guide describes how to perform recurring administrative operations in Web Agent.

Audit the deployment

Web Agent logs audit events for security, troubleshooting, and regulatory compliance.

Remote and local auditing

Remote auditing

In remote auditing, the agent logs events to the audit event handler configured in the AM realm. In an environment with several AM servers, the agent writes audit logs to the AM server that satisfies the agent request for client authentication or resource authorization.

The agent logs audit events remotely only when AM’s global audit logging is enabled and configured in the realm where the agent runs.

Set up global audit logging in the AM admin UI:

  1. In the AM admin UI, go to Configure > Global Services > Audit logging.

  2. Enable Audit logging.

  3. Enter values to include in Field whitelist filters or Field blacklist filters.

The following example path in the Field whitelist filters list includes the Accept-Language value in the http.request.headers field in access events:

/access/http/request/headers/accept-language

Learn more from AM’s Global audit logging.

Local auditing

In local auditing, the agent logs audit events in JSON format to /path/to/web_agents/agent_type/instances/agent_n/logs/audit/audit.log.

An example agent log file is /path/to/web_agents/apache24_agent/instances/agent_1/logs/audit/audit.log.

Remote and local auditing

In remote and local auditing, the agent logs audit events in the following locations:

  • To /path/to/web_agents/agent_type/instances/agent_n/logs/audit/audit.log

  • To the audit event handler configured in the AM realm in which the agent profile is configured.

Audit event logs

Audit logs are written in UTF-8 format. The following example shows an audit event log for successful access to a resource:

{
   "timestamp":"2023-10-30T11:56:57Z",
   "eventName":"AM-ACCESS-OUTCOME",
   "transactionId":"608...77e",
   "userId":"id=demo,ou=user,dc=example,dc=com",
   "trackingIds":[
      "fd5...095",
      "fd5...177"
   ],
   "component":"Web Policy Agent",
   "realm":"/",
   "server":{
      "ip":"127.0.0.1",
      "port":8020
   },
   "request":{
      "protocol":"HTTP/1.1",
      "operation":"GET"
   },
   "http":{
      "request":{
         "secure":false,
         "method":"GET",
         "path":"/examples",
         "cookies":{
            "am-auth-jwt":"eyJ...iOi[...]"
            "i18next":"en",
            "amlbcookie":"01",
            "iPlanetDirectoryPro":"Ts2...oxR[...]"
         }
      }
   },
   "response":{
      "status":"DENIED"
   },
   "_id":"fd5...703" //This ID is internal to AM and available only in remote logs.
}
json

The audit log format uses the log structure shared by the Ping Identity Platform. Learn more from Audit log format in AM’s Security guide.

Web Agent supports propagation of the transaction ID across the Ping Identity Platform, using the HTTP header X-ForgeRock-TransactionId. Learn more from Trust transaction headers in AM’s Security guide.

Configure auditing

By default, auditing is disabled. Configure audit logging as follows:

  1. On the AM admin UI, select Realms > Realm Name > Applications > Agents > Web > Agent Name.

  2. On the Global tab, select the following options to select the type of audit events to log and the audit location. By default, auditing is disabled:

  3. In agent.conf, optionally configure Audit Path as Full URL to log the full URL of the HTTP request. If not configured, only the path component of the HTTP request is logged.

  4. In agent.conf, optionally configure the following properties to manage the location and size of the log files:

    After changing a bootstrap property, restart the web server where the agent runs.

Monitor services

The following sections describe how to set up and maintain monitoring in your deployment to ensure appropriate performance and service availability.

Monitor with Prometheus

Web Agent automatically exposes a monitoring endpoint where Prometheus can scrape metrics in a standard Prometheus format (version 0.0.4).

You can find information about installing and running Prometheus in the Prometheus documentation.

The Prometheus endpoint is protected by HTTP Basic Authentication. To access it, provide the agent URL, and the agent profile name and password. Always use HTTPS for secure connections to client applications.

The metrics returned are described in Metrics at the Prometheus endpoint.

Tools such as Grafana are available to create customized charts and graphs based on the information collected by Prometheus. Learn more on the Grafana website.

Access the Prometheus endpoint

  1. Install a Web Agent as described in the Installation, and use the agent to protect a web application. For example, set up the example in Policy enforcement.

  2. Access the Prometheus endpoint as follows, where https://agent.example.com:443 is the agent URL, web-agent is the agent profile name and password is the agent profile password:

    $ curl https://agent.example.com:443/agent/metrics --user web-agent:password

    The metrics are displayed:

    # TYPE policy_change counter
# HELP policy_change_total number of policy updates
policy_change_total{topic="notification"} 0
# TYPE config_change counter
# HELP config_change_total number of configuration changes
config_change_total{topic="notification"} 0
# TYPE not_enforced counter
# HELP not_enforced_total number of requests that were not enforced
not_enforced_total{topic="enforcement"} 0
...

Monitoring types

This section describes the data types used in monitoring:

Counter

Cumulative metric for a numerical value that only increases.

Gauge

Metric for a numerical value that can increase or decrease.

The value for a gauge is calculated when requested and represents the state of the metric at that specific time.

Histogram

Metric that samples observations, counts them in buckets, and provides a sum of all observed values.

Metrics at the Prometheus endpoint

Notification metrics

Web Agent exposes the following notification-related monitoring metrics:

Metric Type Description

policy_change_total

Counter

Number of policy change notifications received from Advanced Identity Cloud or AM.

config_change_total

Counter

Number of agent configuration change notifications received from Advanced Identity Cloud or AM.

Policy decision metrics

Web Agent exposes the following policy decision monitoring metrics:

Metric Type Description

not_enforced_total

Counter

Number of requests that weren’t enforced by the agent because of the not-enforced URL lists.

not_authorised_total

Counter

Number of requests denied by policy.

not_authenticated_total

Counter

Number of requests requiring authentication.

local_decision_total

Counter

Number of policy decisions the agent makes locally.

remote_decision_total

Counter

Number of policy decisions the agent requests from Advanced Identity Cloud or AM.

cache_decision_total

Counter

Number of policy decisions the agent takes from the cache.

Cache metrics

Web Agent exposes the following cache-related monitoring metrics:

Metric Type Description

cache_write_total

Counter

Number of session cache writes.

cache_update_total

Counter

Number of session cache updates.

cache_read_total

Counter

Number of session cache reads.

cache_miss_total

Counter

Number of sessions not found in cache.

cache_delete_total

Counter

Number of sessions deleted from cache.

cache_expiry_total

Counter

Number of sessions expired from cache.

cache_fault_total

Counter

Number of sessions that couldn’t be cached.

cache_occupancy

Gauge

Proportion of session cache that is occupied.

Connection metrics

Web Agent exposes the following connection-related monitoring metrics:

Metric Type Description

connection_total

Counter

Number of connections created.

connection_reuse_total

Counter

Number of cached connections reused.

Request metrics

Web Agent exposes the following request monitoring metrics:

Metric Type Description

policy_request_seconds

Histogram

Histogram of policy request times in seconds.

session_request_seconds

Histogram

Histogram of session request times in seconds.

config_request_seconds

Histogram

Histogram of configuration request times in seconds.

agent_time_seconds

Histogram

Histogram of agent time in request pipeline in seconds.

Monitor with the monitoring endpoint (deprecated)

The monitoring endpoint described in this section is deprecated. Use it only for diagnostics, in conjunction with Support.

A monitoring endpoint provides access to metrics for operations within the agent and between the agent an AM.

The monitoring endpoint is protected by HTTP Basic Authentication. To access it, provide the agent URL, and the agent profile name and password. Always use HTTPS for secure connections to client applications.

Metrics are displayed as a JSON response, with the fields described in Metrics at the monitoring endpoint (deprecated).

Access the monitoring endpoint

  1. Install a Web Agent as described in the Installation, and use the agent to protect a web application. For example, set up the example in Policy enforcement.

  2. Access the agent monitoring endpoint as follows, where https://agent.example.com:443 is the agent URL, and web-agent is the agent profile name.

    $ curl https://agent.example.com:443/agent/monitor --user web-agent

Enter host password for user 'web-agent':

  3. Enter the agent profile password to display the metrics:

    {
  "cache-invalidation": {
    "policy": 0,
    "profile": 1
  },
  "policy-decisions": {
    "neu": 0,
    "local": 0,
    "remote": 2,
    "cache": 0
    },
  "gc": {
    "runs": 1,
    "released": 0,
    "release-deferred": 0,
    "fill": 0.000000
  },
  "cache-operations": {
    "writes": 0,
    "rewrites": 2,
    "reads": 2,
    "misses": 0,
    "deletes": 0,
    "write-faults": 0,
    "expired": 0,
    "occupancy": 0
  },
  "connections": {
    "added": 2,
    "reused": 3
  }
}
    json

Metrics at the monitoring endpoint (deprecated)

Metric Submetric Count of

cache-invalidation

policy

Number of policy change notifications received from AM.

profile

Number of agent configuration change notifications received from AM.

policy-decisions

neu

Number of requests that were not enforced by the agent because of the not-enforced URL lists.

local

Number of policy decisions the agent makes locally.

remote

Number of policy decisions the agent requests from AM.

cache

Number of policy decisions the agent takes from the cache.

gc

runs

Number of garbage collection runs.

released

Number of cache entries released during garbage collection runs.

release-defered

Number of entries with release deferred until the next garbage collection run.

fill

Floating point value between 0 and 1, representing the proportion of cache that is free after the most the recent garbage collection.

cache-operations

writes

Number of writes to cache.

rewrites

Number of updates to cache.

reads

Number of reads from cache.

misses

Number of failed searches of the cache.

deletes

Number of deletes from cache.

write-faults

Number of cache writes that fail because the cache is full.

expired

Number of expired cache entries.

occupancy

Proportion of cache that is occupied.

connections

added

Number of new connections made.

reused

Number of times existing connections were reused.

Notifications

AM sends the following notifications to Web Agent through WebSockets:

Configuration notifications

When the administrator makes a change to a hot-swappable agent configuration property, AM sends a notification to the agent to reread the agent profile from AM.

Configuration notifications apply when the agent profile is stored in AM’s configuration data store.

For more information about the cache, refer to Configuration cache.

Session Notifications

When a client logs out, or a CTS-based session expires, AM sends a notification to the agent to remove the client’s entry from the session cache.

For more information about the cache, refer to Session and policy decision cache.

Policy Notifications

When an administrator changes a policy, AM sends a notification to the agent to flush the session and policy decision cache, and the policy cache. Enable Notifications controls whether the AM server sends notifications to connected agents. It is enabled by default.

For more information about the cache, refer to Session and policy decision cache and Policy cache.

In configurations with load balancers and reverse proxies, make sure the load balancers and reverse proxies support WebSockets.

The AM advanced server configuration property, org.forgerock.openam.notifications.agents.enabled, controls whether the AM server sends notifications to connected agents. This property is enabled by default.

Disable notifications

Notifications are enabled by default. Before disabling notifications, consider the impact on security if the agent is not notified of changes in AM.

  1. On the AM admin UI, select Realms > Realm Name > Applications > Agents > Web > Agent Name.

  2. On the Global tab, deselect the following options to disable notifications:

Tune connections

Use a connection pool between Web Agent and AM to cache and reuse connections, and so reduce the overhead of creating new connections. The agent can use an array of connections concurrently, with multiple request threads.

To enable connection pooling, set Enable Connection Pooling to true. Test and tune the performance of your deployment with connection pooling before you use it in a production environment.

The following image shows the architecture of a connection pool:

Connection pool

The following image shows the flow of information when a request is treated in a connection pool:

connection-pool

When a client makes a request, the agent intercepts the request and uses the connection pool to connect to AM. If a connection is available, the agent uses that connection. The client is unaware of the connection reuse.

If a connection is not available, and fewer than 1024 connections are in use, the agent creates and uses a new connection. If 1024 connections are already in use, the request waits until an existing connection is released, or a new connection can be created.

When 1024 connections are in use, the agent creates additional temporary connections. Connections can be closed by AM/IDC, but the agent reopens them when it detects that they are closed.

When the request is complete, the agent closes the connection to the pool, but retains the physical connection. The connection is then available to requests with the same connection parameters.

Consider the following points for connection pooling:

  • The connection pool can contain up to 1024 cached connections

  • When more than 1024 connections are required, the agent creates temporary connection.

  • By default, connections timeout after four seconds of waiting for a response. To change this value, configure Connection Timeout

  • Tune Connection Timeout so that it is:

    • Long enough for systems to respond, and therefore prevent unnecessary failures

    • As short as possible to minimize the time to wait after a network failure

  • To reduce the overhead of making new connections and SSL handshakes, set the HTTP keep-alive headers for AM containers or reverse proxies to longer than Connection Timeout.

Rotate keys

Key rotation is the process of generating a new version of a key, assigning that version, and then deprovisioning the old key.

Why and when to rotate keys

Regular key rotation is a security consideration that is sometimes required for internal business compliance. Regularly rotate keys to:

  • Limit the amount of data protected by a single key.

  • Reduce dependence on specific keys, making it easier to migrate to stronger algorithms.

  • Prepare for when a key is compromised. The first time you try key rotation shouldn’t be during a real-time recovery.

Key revocation is a type of key rotation done exceptionally if you suspect that a key has been compromised. To decide when to revoke a key, consider the following points:

  • If limited use of the old keys can be tolerated, provision the new keys and then deprovision the old keys. Messages produced before the new keys are provisioned are impacted.

  • If use of the old keys can’t be tolerated, deprovision the old keys before you provision the new keys. The system is unusable until new keys are provisioned.

Steps for rotating keys

  1. Stop the web server.

  2. View a list of Web Agent instances, using the agentadmin --l command.

  3. Rotate the keys for a Web Agent instance, using the agentadmin --k --rotate agent-instance command.

    The following example rotates keys for the instance agent_3:

    $ cd /path/to/web_agents/apache24_agent/bin/
$ ./agentadmin --k --rotate agent_3

Performing key rotation for instance: agent_3

Instance config directory: /path/to/web_agents/apache24_agent/instances/agent_3
Loading agent.conf…​done
Loading current credentials…​done
Generating new encryption key…​done
Encrypting current credentials with new encryption key:
	- Encrypting agent profile password with new key…​done
	- Encrypting certificate password with new key…​done
	- Encrypting http proxy password with new key…​done
Performing file operations:
Gathering file information for agent-key.conf
Gathering file information for agent-password.conf
Backing up key file to agent-key.conf.bak
Backing up password file to agent-password.conf.bak
Writing new key to agent-key.conf…​done
Writing new ciphertexts to agent-password.conf…​done
Successfully wrote new key and passwords to disk

Removing backup agent-key.conf.bak…​done
Removing backup agent-password.conf.bak…​done

Key rotation was successful for instance: agent_3

Considerations if key rotation fails

  • If key rotation fails while the agent is updating agent-password.conf or agent-key.conf, the rotate command tries to revert to the original files.

  • If the rotate command can’t revert to the original files, manually move agent-password.conf.bak and agent-key.conf.bak to agent-password.conf and agent-key.conf.

  • After a failed key rotation on Windows, look for and delete .bak files. Windows can’t rename a file as .bak if a .bak file already exists.