Java Agents 2023.11.2

Maintenance guide

This guide describes how to perform recurring administrative operations in ForgeRock Access Management Java Agent.

About ForgeRock Identity Platform™ software

ForgeRock Identity Platform serves as the basis for our simple and comprehensive Identity and Access Management solution. For more information, visit https://www.pingidentity.com.

Auditing

Java Agent logs audit events for security, troubleshooting, and regulatory compliance. Logs are written in UTF-8. Store audit event logs in the following ways:

Remotely

Log audit events to the audit event handler configured in the AM realm. In an environment with several AM servers, agents write audit logs to the AM server that satisfies the agent request for client authentication or resource authorization.

Java Agent cannot log audit events remotely if:

  • AM’s Audit Logging Service is disabled.

  • No audit event handler is configured in the realm where the agent is configured.

  • All audit event handlers configured in the realm where the agent is configured are disabled.

For more information about audit logging in AM, refer to Setting up audit logging in AM’s Security guide.

Locally

Log audit events in JSON format to a file in the agent installation directory, /java_agents/agent_type/logs/audit/ .

Remotely and locally

Log audit events:

  • To a file in the agent installation directory.

  • To the audit event handler configured in the AM realm in which the agent profile is configured.

The following is an example of an agent log record:

{
   "timestamp":"...",
   "eventName":"AM-ACCESS-OUTCOME",
   "transactionId":"608831c4-7351-4277-8a5f-b1a83fe2277e",
   "userId":"id=demo,ou=user,dc=openam,dc=forgerock,dc=org",
   "trackingIds":[
      "fd5c8ccf-7d97-49ba-a775-76c3c06eb933-82095",
      "fd5c8ccf-7d97-49ba-a775-76c3c06eb933-82177"
   ],
   "component":"Java Policy Agent",
   "realm":"/",
   "server":{
      "ip":"127.0.0.1",
      "port":8020
   },
   "client":{
      "ip":"127.0.0.1",
      "port":55180
   },
   "request":{
      "protocol":"HTTP/1.1",
      "operation":"GET"
   },
   "http":{
      "request":{
         "secure":false,
         "method":"GET",
         "path":"http://my.example.com:8020/examples/",
         "headers":{
            "referer":[
               "https://am.example.com:8443/am/oauth2/authorize?scope[...]"
            ],
            "accept-language":[
               "en,en-US;q=0.8,da;q=0.6,fr;q=0.4"
            ],
            "host":[
               "my.example.com:8020"
            ],
            "upgrade-insecure-requests":[
               "1"
            ],
            "connection":[
               "keep-alive"
            ],
            "cache-control":[
               "max-age=0"
            ],
            "accept-encoding":[
               "gzip, deflate"
            ],
            "user-agent":[
               "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko)[...]"
            ],
            "accept":[
               "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
            ]
         },
         "cookies":{
            "am-auth-jwt":"eyJ0eXAiOiJKV1QiLCJhbGciOi[...]"
            "i18next":"en",
            "amlbcookie":"01",
            "iPlanetDirectoryPro":"Ts2zDkGUqgtkoxR[...]"
         }
      }
   },
   "response":{
      "status":"DENIED"
   },
   "_id":"fd5c8ccf-7d97-49ba-a775-76c3c06eb933-81703"
}
Local audit logs do not have an _id attribute, which is an internal AM id.

The audit log format adheres to the log structure shared across the ForgeRock Identity Platform. For more information about the audit log format, refer to Audit log format in AM’s Security guide.

Java Agent supports propagation of the transaction ID across the ForgeRock Identity Platform, using the HTTP header X-ForgeRock-TransactionId. For more information about configuring the header, refer to Configuring the trust transaction header system property in AM’s Security guide.

Configure audit logging

By default, Java Agent does not write audit log records. To configure audit logging, perform this procedure. The agent in this example is in remote configuration mode.

  1. In the AM admin UI, select Realms > realm name > Applications > Agents > Java > agent name > Global > Audit.

  2. In Audit Access Types, select the type of messages to log. For example, select LOG_ALL to log access allowed and access denied events.

  3. In Audit Log Location, select whether to write the audit logs locally to the agent installation (LOCAL), remotely to AM (REMOTE), or to both places (ALL). For example, keep REMOTE to log audit events to the AM instances.

  4. To log audit messages locally, enable Enable Local Audit Log Rotation to rotate the audit log files when they reach a maximum size.

  5. If you enabled Enable Local Audit Log Rotation, specify the maximum size of the audit log files in Local Audit Log Rotation Size.

Monitor services

The following sections describe how to set up and maintain monitoring in your deployment to ensure appropriate performance and service availability.

The Prometheus endpoint isn’t currently supported for Jakarta builds of the agent because of known issue AMAGENTS-6809.

Monitor with Prometheus

Java Agent automatically exposes a monitoring endpoint where Prometheus can scrape metrics in a standard Prometheus format (version 0.0.4).

The Prometheus endpoint is enabled by default. You can disable it if required by setting the Enable Prometheus Monitoring property to false.

You can find information about installing and running Prometheus in the Prometheus documentation.

Prometheus performance metrics are provided by an endpoint configured in the protected web application’s web.xml file. The endpoint must be accessible to the Prometheus server that uses the performance data.

Tools such as Grafana are available to create customized charts and graphs based on the information collected by Prometheus. Learn more on the Grafana website.

Expose the Prometheus endpoint

Use the following procedure to expose the Prometheus endpoint.

  1. For each protected web application that is to expose metrics, edit the web.xml file for the agentapp application.

    The following example exposes a base endpoint named /metrics:

    <servlet>
   <servlet-name>AgentMonitoring</servlet-name>
   <servlet-class>org.forgerock.http.servlet.HttpFrameworkServlet</servlet-class>
   <init-param>
       <param-name>application-loader</param-name>
       <param-value>service_loader</param-value>
   </init-param>
</servlet>
<servlet-mapping>
    <servlet-name>AgentMonitoring</servlet-name>
    <url-pattern>/metrics/*</url-pattern>
</servlet-mapping>

    Choose a name for the exposed base endpoint that does not conflict with any of the built-in agent endpoints.

  2. Allow access to the Prometheus endpoint that is protected by the agent, in one of the following ways:

    • Configure Not-Enforced URIs to create a not-enforced URI rule for the base endpoint.

      The following example rule allows access to the metrics base endpoint:

      */metrics/*

    • Configure Not-Enforced URIs, Not-Enforced Client IP List, and Not-Enforced Compound Rule Separator to create a compound not-enforced rule for the base endpoint.

      The rule allows access from only the IP addresses of the Prometheus server.

      The following example rule allows access to the /metrics endpoint for HTTP requests that come from the IP address range from 192.168.1.1 to 192.168.1.3:

      192.168.1.1-192.168.1.3 | */metrics/*

      HTTP requests from other IP addresses are not able to access the metrics base endpoint.

    • Create an authorization policy in Identity Cloud or AM to restrict access to the metrics base endpoint.

      The metric base endpoint doesn’t require login credentials. You can use a policy to ensure that requests to the endpoint are authenticated against the Identity Cloud or AM instance.

      Learn more in Policies in Identity Cloud’s Authorization guide or Policies in AM’s Authorization guide.

  3. If the Prometheus endpoint is protected by Identity Cloud or AM policies, include the required credentials.

Access the Prometheus endpoint

When the example in Expose the Prometheus endpoint is configured, the Prometheus endpoint is available at https://mydomain.example.com/agentapp/metrics/prometheus.

  1. Access the Prometheus endpoint as follows:

    $ curl https://mydomain.example.com/agentapp/metrics/prometheus

    The metrics are displayed:

    # HELP ja_expired_session_cache_eviction the eviction count for the expired session cache
# TYPE ja_expired_session_cache_eviction gauge
ja_expired_session_cache_eviction 0.0
# HELP ja_expired_session_cache the hit count for the expired session cache
# TYPE ja_expired_session_cache gauge
ja_expired_session_cache{outcome="hit",} 0.0
ja_expired_session_cache{outcome="miss",} 0.0
# HELP ja_expired_session_cache_load the total load count for the expired session cache
# TYPE ja_expired_session_cache_load gauge
...

Write metrics to CSV files

Configure Export Monitoring Metrics to CSV to write metric information to CSV files.

Monitoring types

This section describes the data types used in monitoring:

Counter

Cumulative metric for a numerical value that only increases.

Distinct counter

Metric that provides an estimate of the number of unique values recorded.

For example, use this to estimate the number of unique users who have authenticated, or unique client IP addresses.

The DistinctCounter type is reported as a Gauge type.

The DistinctCounter metric is calculated per instance of AM and can’t be aggregated across multiple instances to get a site-wide view.
Gauge

Metric for a numerical value that can increase or decrease.

The value for a gauge is calculated when requested and represents the state of the metric at that specific time.

Timer

Metric that provides duration information.

The Prometheus endpoint doesn’t provide rate-based statistics because rates can be calculated from the time-series data. Instead, the Prometheus endpoint includes summary metrics whose names have the following suffixes or labels:

  • count: number of events recorded

  • sum: sum of the durations recorded

  • {quantile="0.5"}: 50% of the durations are at or below this value

  • {quantile="0.75"}: 75% of the durations are at or below this value

  • {quantile="0.95"}: 95% of the durations are at or below this value

  • {quantile="0.98"}: 98% of the durations are at or below this value

  • {quantile="0.99"}: 99% of the durations are at or below this value

  • {quantile="0.999"}: 99.9% of the durations are at or below this value

The Timer metric type is reported as a Summary type.

Duration-based quantile values are weighted towards newer data. By representing approximately the last five minutes of data, the timers make it easier to see recent changes in behavior, rather than a uniform average of recordings since the server was started.

Metrics at the Prometheus endpoint

Audit handler metrics

Java Agent exposes the following audit handler monitoring metrics:

Metric Type Description

ja_audit_generate{topic=access}

Timer

Time taken to generate an audit object.

ja_audit{handler-type=handler-type,name=default,topic=access,outcome=outcome}

Timer

Time taken to audit outcomes, both locally to the agent and remotely in Identity Cloud or AM.
Audit handler metrics labels
Label Values

handler-type

  • am-delegate. Remote auditing performed by Identity Cloud or AM.

    (Prometheus: am_delegate)

  • json. Local audit logging using JSON.

outcome

  • success

  • failure

Endpoint and REST SDK metrics

Java Agent exposes the following endpoint and REST SDK monitoring metrics:

Metric Type Description

ja_session_info

Timer

Time taken to retrieve user session information from Identity Cloud or AM.

ja_user_profile

Timer

Time taken to retrieve the user profile information from Identity Cloud or AM.

ja_policy_decision

Timer

Time taken to retrieve policy decisions from Identity Cloud or AM.

Expired session metrics

Java Agent exposes the following expired session monitoring metrics:

Metric Type Description

ja_expired_session_cache_size

Gauge

Total number of entries in the expired session cache.

ja_expired_session_cache_eviction

Gauge

Eviction count for the expired session cache.

ja_expired_session_cache_load

Gauge

Load count for the expired session cache.

ja_expired_session_cache_load_time

Gauge

Load time for the expired session cache, in milliseconds.

ja_expired_session_cache{outcome=hit}

Gauge

Hit count for the expired session cache.

ja_expired_session_cache{outcome=miss}

Gauge

Miss count for the expired session cache.

JSON Web Token (JWT) metrics

Java Agent exposes the following JWT-related monitoring metrics:

Metric Type Description

ja_jwt_cache_size

Gauge

Size of the JWT cache.

ja_jwt_cache_eviction

Gauge

The eviction count for the JWT cache.

ja_jwt_cache_load

Gauge

The load count for the JWT cache.

ja_jwt_cache_load_time

Gauge

The load time for the JWT cache, in milliseconds.

ja_jwt_cache{outcome=hit}

Gauge

The hit count for the JWT cache.

ja_jwt_cache{outcome=miss}

Gauge

The miss count for the JWT cache.

JVM metrics

Java Agent exposes the following JVM-related monitoring metrics:

Some ja_jvm_* metrics depend on the JVM version and configuration. In particular, GC-related metrics depend on the garbage collector the server uses. The GC metric names are unstable, and can change even in a minor JVM release.
Metric Type Description

ja_jvm_available_cpus

Gauge

Number of processors available to the Java virtual machine.

ja_jvm_class_loading_loaded

Gauge

Number of classes loaded since the Java virtual machine started.

ja_jvm_class_loading_unloaded

Gauge

Number of classes unloaded since the Java virtual machine started.

ja_jvm_free_used_memory_bytes

Gauge

Bytes of free memory allocated to the Java virtual machine

ja_jvm_garbage_collector_g1_old_generation_collection

Gauge

Number of collections performed by the "G1 old generation" garbage collection algorithm.

ja_jvm_garbage_collector_g1_old_generation_time

Gauge

Approximate accumulated time taken by the "G1 old generation" garbage collection algorithm.

ja_jvm_garbage_collector_g1_young_generation_collection

Gauge

Number of collections performed by the "G1 young generation" garbage collection algorithm.

ja_jvm_garbage_collector_g1_young_generation_time

Gauge

Approximate accumulated time taken by the "G1 young generation" garbage collection algorithm.

ja_jvm_max_memory_bytes

Gauge

Maximum amount of memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_heap_init

Gauge

Amount of heap memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_heap_max

Gauge

Maximum amount of heap memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_heap_committed

Gauge

Amount of heap memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_heap_used

Gauge

Amount of heap memory used by the Java virtual machine.

ja_jvm_memory_usage_heap_usage

Gauge

Amount of heap memory allocated to the Java virtual machine.

ja_jvm_memory_usage_total_init

Gauge

Amount of memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_total_max

Gauge

Maximum amount of memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_non_heap_init

Gauge

Amount of non-heap memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_non_heap_max

Gauge

Maximum amount of non-heap memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_non_heap_committed

Gauge

Amount of non-heap memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_non_heap_used

Gauge

Amount of non-heap memory used by the Java virtual machine.

ja_jvm_memory_usage_non_heap_usage

Gauge

Amount of non-heap memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_codeheap_non_nmethods_committed

Gauge

Amount of "non-method code cache heap" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_codeheap_non_nmethods_init

Gauge

Amount of "non-method code cache heap" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_codeheap_non_nmethods_max

Gauge

Maximum amount of "non-method code cache heap" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_codeheap_non_nmethods_usage

Gauge

Amount of "non-method code cache heap" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_codeheap_non_nmethods_used

Gauge

Amount of "non-method code cache heap" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_codeheap_non_profiled_nmethods_committed

Gauge

Amount of "non-profiled code cache heap" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_codeheap_non_profiled_nmethods_init

Gauge

Amount of "non-profiled code cache heap" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_codeheap_non_profiled_nmethods_max

Gauge

Maximum amount of "non-profiled code cache heap" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_codeheap_non_profiled_nmethods_usage

Gauge

Amount of "non-profiled code cache heap" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_codeheap_non_profiled_nmethods_used

Gauge

Amount of "non-profiled code cache heap" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_codeheap_profiled_nmethods_committed

Gauge

Amount of "profiled code cache heap" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_codeheap_profiled_nmethods_init

Gauge

Amount of "profiled code cache heap" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_codeheap_profiled_nmethods_max

Gauge

Maximum amount of "profiled code cache heap" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_codeheap_profiled_nmethods_usage

Gauge

Amount of "profiled code cache heap" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_codeheap_profiled_nmethods_used

Gauge

Amount of "profiled code cache heap" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_compressed_class_space_init

Gauge

Amount of "compressed class space" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_compressed_class_space_max

Gauge

Maximum amount of "compressed class space" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_compressed_class_space_committed

Gauge

Amount of "compressed class space" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_compressed_class_space_used

Gauge

Amount of "compressed class space" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_compressed_class_space_usage

Gauge

Amount of "compressed class space" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_g1_eden_space_committed

Gauge

Amount of "G1 eden space" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_g1_eden_space_init

Gauge

Amount of "G1 eden space" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_g1_eden_space_max

Gauge

Maximum amount of "G1 eden space" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_g1_eden_space_usage

Gauge

Amount of "G1 eden space" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_g1_eden_space_used

Gauge

Amount of "G1 eden space" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_g1_eden_space_used_after_gc

Gauge

Amount of "G1 eden space" memory used by the Java virtual machine after garbage collection.

ja_jvm_memory_usage_pools_g1_old_gen_committed

Gauge

Amount of "G1 old generation" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_g1_old_gen_init

Gauge

Amount of "G1 old generation" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_g1_old_gen_max

Gauge

Maximum amount of "G1 old generation" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_g1_old_gen_usage

Gauge

Amount of "G1 old generation" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_g1_old_gen_used

Gauge

Amount of "G1 old generation" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_g1_old_gen_used_after_gc

Gauge

Amount of "G1 old generation" memory used by the Java virtual machine after garbage collection.

ja_jvm_memory_usage_pools_g1_survivor_space_committed

Gauge

Amount of "G1 survivor space" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_g1_survivor_space_init

Gauge

Amount of "G1 survivor space" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_g1_survivor_space_max

Gauge

Maximum amount of "G1 survivor space" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_g1_survivor_space_usage

Gauge

Amount of "G1 survivor space" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_pools_g1_survivor_space_used

Gauge

Amount of "G1 survivor space" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_g1_survivor_space_used_after_gc

Gauge

Amount of "G1 survivor space" memory used by the Java virtual machine after garbage collection.

ja_jvm_memory_usage_pools_metaspace_init

Gauge

Amount of "metaspace" memory the Java virtual machine initially requested from the operating system.

ja_jvm_memory_usage_pools_metaspace_max

Gauge

Maximum amount of "metaspace" memory the Java virtual machine attempts to use.

ja_jvm_memory_usage_pools_metaspace_committed

Gauge

Amount of "metaspace" memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_pools_metaspace_used

Gauge

Amount of "metaspace" memory used by the Java virtual machine.

ja_jvm_memory_usage_pools_metaspace_usage

Gauge

Amount of "metaspace" memory allocated to the Java virtual machine.

ja_jvm_memory_usage_total_committed

Gauge

Amount of memory committed for the Java virtual machine to use.

ja_jvm_memory_usage_total_used

Gauge

Amount of memory used by the Java virtual machine.

ja_jvm_thread_state_blocked_result

Gauge

Number of threads in the BLOCKED state.

ja_jvm_thread_state_result

Gauge

Number of live threads including both daemon and non-daemon threads.

ja_jvm_thread_state_daemon_result

Gauge

Number of live daemon threads.

ja_jvm_thread_state_new_result

Gauge

Number of threads in the NEW state.

ja_jvm_thread_state_runnable_result

Gauge

Number of threads in the RUNNABLE state.

ja_jvm_thread_state_terminated_result

Gauge

Number of threads in the TERMINATED state.

ja_jvm_thread_state_timed_waiting_result

Gauge

Number of threads in the TIMED_WAITING state.

ja_jvm_thread_state_waiting_result

Gauge

Number of threads in the WAITING state.

ja_jvm_used_memory_bytes

Gauge

Amount of memory used by the Java virtual machine

Not-enforced rule metrics

Java Agent exposes the following not-enforced rule monitoring metrics:

Metric Type Description

ja_notenforced_uri_matched_cache_size

Gauge

Size of the not-enforced URI matched cache.

ja_notenforced_uri_matched_cache_eviction

Gauge

Eviction count for the not-enforced URI matched cache.

ja_notenforced_uri_matched_cache_load

Gauge

Load count for the not-enforced URI matched cache.

ja_notenforced_uri_matched_cache_load_time

Gauge

Load time for the not-enforced URI matched cache, in milliseconds.

ja_notenforced_uri_matched_cache{outcome=hit}

Gauge

Hit count for the not-enforced URI matched cache.

ja_notenforced_uri_matched_cache{outcome=miss}

Gauge

Miss count for the not-enforced URI matched cache.

ja_notenforced_uri_unmatched_cache_size

Gauge

Size of the not-enforced URI unmatched cache.

ja_notenforced_uri_unmatched_cache_eviction

Gauge

Eviction count for the not-enforced URI unmatched cache.

ja_notenforced_uri_unmatched_cache_load

Gauge

Load count for the not-enforced URI unmatched cache.

ja_notenforced_uri_unmatched_cache_load_time

Gauge

Load time for the not-enforced URI unmatched cache, in milliseconds.

ja_notenforced_uri_unmatched_cache{outcome=hit}

Gauge

Hit count for the not-enforced URI unmatched cache.

ja_notenforced_uri_unmatched_cache{outcome=miss}

Gauge

Miss count for the not-enforced URI unmatched cache.

ja_notenforced_ip_matched_cache_size

Gauge

Size of the not-enforced IP matched cache.

ja_notenforced_ip_matched_cache_eviction

Gauge

Eviction count for the not-enforced IP matched cache.

ja_notenforced_ip_matched_cache_load

Gauge

Load count for the not-enforced IP matched cache.

ja_notenforced_ip_matched_cache_load_time

Gauge

Load time for the not-enforced IP matched cache, in milliseconds.

ja_notenforced_ip_matched_cache{outcome=hit}

Gauge

Hit count for the not-enforced IP matched cache.

ja_notenforced_ip_matched_cache{outcome=miss}

Gauge

Miss count for the not-enforced IP matched cache.

ja_notenforced_ip_unmatched_cache_size

Gauge

Size of the not-enforced IP unmatched cache.

ja_notenforced_ip_unmatched_cache_eviction

Gauge

Eviction count for the not-enforced IP unmatched cache.

ja_notenforced_ip_unmatched_cache_load

Gauge

Load count for the not-enforced IP unmatched cache.

ja_notenforced_ip_unmatched_cache_load_time

Gauge

Load time for the not-enforced IP unmatched cache, in milliseconds.

ja_notenforced_ip_unmatched_cache{outcome=hit}

Gauge

Hit count for the not-enforced IP unmatched cache.

ja_notenforced_ip_unmatched_cache{outcome=miss}

Gauge

Miss count for the not-enforced IP unmatched cache.

ja_notenforced_compound_matched_cache_size

Gauge

Size of the not-enforced compound matched cache.

ja_notenforced_compound_matched_cache_eviction

Gauge

Eviction count for the not-enforced compound matched cache.

ja_notenforced_compound_matched_cache_load

Gauge

Load count for the not-enforced compound matched cache.

ja_notenforced_compound_matched_cache_load_time

Gauge

Load time for the not-enforced compound matched cache, in milliseconds.

ja_notenforced_compound_matched_cache{outcome=hit}

Gauge

Hit count for the not-enforced compound matched cache.

ja_notenforced_compound_matched_cache{outcome=miss}

Gauge

Miss count for the not-enforced compound matched cache.

ja_notenforced_compound_unmatched_cache_size

Gauge

Size of the not-enforced compound unmatched cache.

ja_notenforced_compound_unmatched_cache_eviction

Gauge

Eviction count for the not-enforced compound unmatched cache.

ja_notenforced_compound_unmatched_cache_load

Gauge

Load count for the not-enforced compound unmatched cache.

ja_notenforced_compound_unmatched_cache_load_time

Gauge

Load time for the not-enforced compound unmatched cache, in milliseconds.

ja_notenforced_compound_unmatched_cache{outcome=hit}

Gauge

Hit count for the not-enforced compound unmatched cache.

ja_notenforced_compound_unmatched_cache{outcome=miss}

Gauge

Miss count for the not-enforced compound unmatched cache.

Policy decision metrics

Java Agent exposes the following policy decision monitoring metrics:

Metric Type Description

ja_policy_decision_cache_size

Gauge

Size of the policy decision cache.

ja_policy_decision_cache_eviction

Gauge

Eviction count for the policy decision cache.

ja_policy_decision_cache_load

Gauge

Load count for the policy decision cache.

ja_policy_decision_cache_load_time

Gauge

Load time for the policy decision cache, in milliseconds.

ja_policy_decision_cache{outcome=hit}

Gauge

Hit count for the policy decision cache.

ja_policy_decision_cache{outcome=miss}

Gauge

Miss count for the policy decision cache.

POST data preservation metrics

Java Agent exposes the following POST data preservation monitoring metrics:

Metric Type Description

ja_pdp_cache_size

Gauge

Size of the POST data preservation cache.

ja_pdp_cache_eviction

Gauge

Eviction count for the POST data preservation cache.

ja_pdp_cache_load

Gauge

Load count for the POST data preservation cache.

ja_pdp_cache_load_time

Gauge

Load time for the POST data preservation cache, in milliseconds.

ja_pdp_cache{outcome=hit}

Gauge

Hit count for the POST data preservation cache.

ja_pdp_cache{outcome=miss}

Gauge

Miss count for the POST data preservation cache.

Request metrics

Java Agent exposes the following request monitoring metrics:

Metric Type Description

ja_requests{access=access,decision=decision}

Timer

Rate of granted/denied requests and their decision.
Request metrics labels
Label Values

access

  • granted

  • denied

decision

  • not-enforced: Request matched a not-enforced rule.

  • no-valid-token: Request didn’t have a valid SSO token or an OpenID Connect JWT.

  • allowed-by-policy: Request matched a policy that allowed access.

  • denied-by-policy: Request matched a policy that denied access.

  • am-unavailable: The Identity Cloud or AM instance wasn’t reachable.

  • bad-request: The request URI was badly formed.

  • unexpected-exception: An unexpected exception occurred within the agent.

Session information metrics

Java Agent exposes the following session information monitoring metrics:

Metric Type Description

ja_session_info_cache_size

Gauge

Size of the session information cache.

ja_session_info_cache_eviction

Gauge

Eviction count for the session information cache.

ja_session_info_cache_load

Gauge

Load count for the session information cache.

ja_session_info_cache_load_time

Gauge

Load time for the session information cache, in milliseconds.

ja_session_info_cache{outcome=hit}

Gauge

Hit count for the session information cache.

ja_session_info_cache{outcome=miss}

Gauge

Miss count for the session information cache.

SSO token to JWT exchange metrics

Java Agent exposes the following SSO token to JWT exchange monitoring metrics:

Metric Type Description

ja_sso_exchange_cache_size

Gauge

Size of the SSO token exchange cache.

ja_sso_exchange_cache_eviction

Gauge

Eviction count for the SSO token exchange cache.

ja_sso_exchange_cache_load

Gauge

Load count for the SSO token exchange cache.

ja_sso_exchange_cache_load_time

Gauge

Load time for the SSO token exchange, in milliseconds.

ja_sso_exchange_cache{outcome=hit}

Gauge

Hit count for the SSO token exchange cache.

ja_sso_exchange_cache{outcome=miss}

Gauge

Miss count for the SSO token exchange cache.

WebSocket metrics

Java Agent exposes the following WebSocket-related monitoring metrics:

Metric Type Description

ja_websocket_last_received

Gauge

Number of milliseconds since anything was received over the WebSocket, for example, a ping or a notification.

ja_websocket_last_sent

Gauge

Number of milliseconds since anything was sent over the WebSocket.

ja_websocket_config_change_received_total

Counter

Count of WebSocket configuration change notifications received.(1)

ja_websocket_config_change_processed_total

Count of WebSocket configuration change notifications processed.

ja_websocket_policy_change_received_total

Counter

Count of WebSocket policy change notifications received.(1)

ja_websocket_policy_change_processed_total

Counter

Count of WebSocket policy change notifications processed.

ja_websocket_session_logout_received_total

Counter

Count of WebSocket session logout notifications received.(1)

ja_websocket_session_logout_processed_total

Counter

Count of WebSocket session logout notifications processed.

ja_websocket_ping_pong

Timer

Ping/pong round trip time.

(1) Some can be ignored if the realm or agent name isn’t applicable.

Logging

Logs in Java Agent and third-party dependencies are recorded using the Logback implementation of the Simple Logging Facade for Java (SLF4J) API.

By default, the logback configuration is stored in /path/to/java_agents/agent_type/Agent_n/config/agent-logback.xml. For information about how to change the location, refer to Agent configuration.

The agent supports the following log levels:

  • TRACE

  • INFO

  • WARN

  • ERROR

  • ON (deprecated, this setting uses the TRACE log level)

  • OFF (deprecated, this setting uses the ERROR log level)

For a full description of logging options, refer to the Logback website.

Setting the log level

The agent maintains the last log level it reads from either agent-logback.xml or Agent Debug Level.

To debug code during the property-reading phase of startup, set the log level in agent-logback.xml to TRACE.

Remote configuration mode

In remote configuration mode, the agent sets the log level as follows:

  • At start up, the agent reads agent-logback.xml, and uses the configured log level.

  • When the agent receives the first request, it reads AgentBootstrap.properties. If org.forgerock.agents.debug.level specifies a different log level, the agent changes the log level.

  • The agent then reads the AM properties. If Agent Debug Level specifies different log level, the agent changes the log level.

  • While the agent is running, if either of the following events occurs the agent changes the log level to the value configured by the event:

Local configuration mode

In local configuration mode, the agent sets the log level as follows:

  • At start up, the agent reads agent-logback.xml, and uses the configured log level.

  • When the agent receives the first request, it reads AgentBootstrap.properties and then AgentConfiguration.properties.

    • If either file specifies a log level in org.forgerock.agents.debug.level, the agent uses that log level.

    • If both files specify a log level, the agent uses the last log level it reads, from AgentConfiguration.properties.

  • While the agent is running, if either of the following events occurs the agent changes the log level to the value configured by the event:

Default logging in agent-logback.xml

By default, agent-logback.xml is configured as follows:

  • When the agent starts, log messages are recorded for the agent.

  • Log messages are written to a text file in /path/to/java_agents/agent_type/Agent_n/logs/debug, where /path/to/java_agents/agent_type/Agent_n is the installation directory.

  • The log level is ERROR.

  • Logs are rolled over once a day, or more frequently if the file reaches 100 MB.

All aspects of the logs are defined by the following reference agent-logback.xml deployed by the agent installer:

<configuration debug="true" scan="true" scanPeriod="60 seconds">
    <appender name="ROLLING" class="org.forgerock.agents.shaded.ch.qos.logback.core.rolling.RollingFileAppender">

        <file>/full/path/to/logs/debug/directory/log.txt</file>
        <rollingPolicy class="org.forgerock.agents.shaded.ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">

            <!-- rollover daily -->
            <fileNamePattern>log-%d{yyyy-MM-dd}.%i.txt</fileNamePattern>

            <!-- each file should be at most 100MB, keep 30 days worth of history, but at most 10GB -->
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%level %msg%n</pattern>
        </encoder>
    </appender>

    <root level="ERROR">
        <appender-ref ref="ROLLING" />
    </root>
</configuration>

Changing agent-logback.xml

To change the logging behavior, update agent-logback.xml.

To take the change into account, stop and restart the agent, or include a scan and an interval configuration in agent-logback.xml. In the following example, agent-logback.xml is scanned every 60 seconds:

<configuration scan="true" scanPeriod="60 seconds" ... >
  ...

For a full description of the options for logging, refer to the Logback website.

Use shaded Logback classnames, to ensure that the agent uses a unique classname compared to other deployed filters and web applications in the container. For example, use

org.forgerock.agents.shaded.ch.qos.logback.core.rolling.RollingFileAppender

instead of

ch.qos.logback.core.rolling.RollingFileAppender

Managing errors in agent-logback.xml

If agent-logback.xml is removed and not replaced, or if a replacement agent-logback.xml contains errors, logs are written to the standard output for the container. For example, Tomcat logs are written to catalina.out.

To log an error when agent-logback.xml fails to load, include a debug configuration:

<configuration debug="true" ... >
  ...

When agent-logback.xml contains errors, messages like these written to the standard output for the container.

ERROR in ch.qos.logback.core.joran.spi.Interpreter@20:72 …​
ERROR in ch.qos.logback.core.joran.action.AppenderRefAction …​

Changing the log level in agent-logback.xml

The global log level is set by default to ERROR by the following line of the default agent-logback.xml:

<root level="ERROR">

To change the global log level, configure the same line with another log level and take the changed file into account, as described in Changing agent-logback.xml.

To change the log level for a single object type without changing it for the rest of the configuration, add a logger defined by the shaded classname of the object, and set its log level.

The following line in agent-logback.xml changes the log level for evaluating not-enforced rules, but does not change the log level of other classes or packages:

<logger name="org.forgerock.agents.shaded.com.sun.identity.agents.common.NotEnforcedRuleHelper" level="TRACE" />

Changing the character set and format of log messages

By default, logs use the system default character set where the agent is running. To use a different character set, or change the pattern of the log messages, edit agent-logback.xml to change the encoder configuration.

The following lines add the date to log messages, and change the character set:

<encoder>
   <pattern>%d{yyyyMMdd-HH:mm:ss} | %-5level | %thread | %logger{20} | %message%n%xException</pattern>
   <charset>UTF-8</charset>
</encoder>

Limiting repeated log messages

To keep log files clean and readable, and to prevent log overflow attacks, limit the number of repeat log messages. Add a custom agent-logback.xml with a DuplicateMessageFilter. This filter detects duplicate messages, and after the specified number of repetitions, drops repeated messages.

The following example allows 5 repetitions of a log message, holds the following 10 repeated messages in the cache, and then discards subsequent repeated messages:

<turboFilter class="org.forgerock.agents.shaded.ch.qos.logback.classic.turbo.DuplicateMessageFilter" allowedRepetitions="5" CacheSize="10" />

The DuplicateMessageFilter has the following limitations:

  • Filters out all duplicate messages. It does not filter per logger, or logger instance, or logger name.

  • Detects repetition of raw messages, meaning that some non-identical messages are considered as repetition.

    For example, when the agent is running at TRACE level, the following message is written to the logs every 20 seconds when there is no other activity over the Websocket:

    TRACE 08:22:00 (20) Sending ping to check connection is still alive

    This message is produced by the logging statement:

    logger.trace("{} ({}) Sending ping to check connection is still alive", timeStamp, lastSent.getSeconds());

    Because the message text {} ({}) Sending ping to check connection is still alive is always the same, it is considered as repeat. Only the first 5 repeats are logged.

  • Does not limit the lifespan of the cache. After the specified number of repetitions is reached, the repeated log messages never appear again, even if they are frequently hit.

Changing the log level in the AM admin UI

To change the log level in the AM admin UI

  1. In the AM admin UI, go to Realms > realm name > Applications > Agents > Java, and select your Java Agent.

  2. On the Global tab, select Agent Debug Level, and select a level.

Alternatively, set Agent Debug Level as a custom property.

Notifications

AM sends the following notifications to Java Agent through WebSockets:

Configuration notifications

When the administrator makes a change to an agent configuration property, AM sends a notification to the agent. The agent flushes the configuration cache, and rereads the agent profile from AM. For more information about the cache, refer to Configuration cache.

Configuration notifications apply when you store the agent profile in AM’s configuration data store. For information, refer to Enable Notifications of Agent Configuration Change.

Session notifications

When a client logs out, or a CTS-based session expires, AM sends a notification to the agent to remove that entry from the session cache. For information, refer to Enable Notification of Session Logout. For more information about the cache, refer to Session Cache.

Policy notifications

When an administrator changes a policy, AM sends a notification to the agent to flush the policy cache. For information, refer to Enable Notification of Policy Changes. For more information about the cache, refer to Policy Cache.

Notifications are enabled by default. To disable notifications, unsubscribe separately to each type of notification. If Autonomous mode is true, notifications and many other features are automatically disabled.

In configurations with load balancers and reverse proxies, make sure that the load balancers and reverse proxies support WebSockets.

For more information about properties that configure notifications, refer to Notifications in the Properties reference.

Tuning connections

Use a connection pool between Java Agent and AM to reuse connections, and so reduce the overhead of creating new connections.

Connection pooling can improve performance in some circumstances. Test and tune the performance of your deployment with connection pooling before you use it in a production environment.

The following image shows the architecture of a connection pool:

Connection pool

When a client makes a request, the agent intercepts the request, and uses the connection pool to connect to AM. If a connection is available, the agent uses that connection. The client is unaware of the connection reuse.

If a connection is not available, and the maximum number of connections is not reached, the agent creates and uses a new connection. When the maximum number of allowed connections is reached, the request waits until an existing connection is released, or a new connection can be created.

When the request is complete, the agent closes the connection to the pool but the physical connection is retained by the pool. The connection is then available to requests with the same connection parameters.

The following image shows the flow of information when a request is treated in a connection pool:

connection-pool

When Enable Connection Pooling is true, use the following properties to tune the connection pool:

Property Description

Max HTTP Connection Count

By default, the connection pool can contain up to 1000 HTTP connections. Change the value of this property according to the AM load, as follows:

  • If AM is likely to be overloaded, reduce the maximum number of connections to reduce the load on AM.

  • If AM is not likely to be overloaded, use the default or a higher value so that connections are available when they are requested.

HTTP Connection Timeout

By default, connections wait for up to 10 seconds for a response before they are considered stale.

Tune this property to allow enough time for systems to respond, and therefore prevent unnecessary failures, but to minimize the time to wait after a network failure.

HTTP Socket Timeout

By default, when a continuous data flow between the connection pool and AM is interrupted, the socket is considered stale.

Tune this property to allow enough time for systems to respond, and therefore prevent unnecessary failures, but to minimize the time to wait after a socket failure.

Enable HTTP Connection Reuse

By default, after a request is complete the connection is returned to the connection pool. The physical connection is retained and can be reused by another request with the same connection parameters.

Reuse connections to reduce the overhead of creating a new connection each time one is requested.

If you are using the connection pool only as a throttling mechanism, to limit the total number of simultaneous connections, it is not necessary to reuse connections.

Enable HTTP Connection State

When Enable HTTP Connection Reuse is true, connection reuse is enabled by default for Apache HTTP Client.

Enable HTTP Retry

By default, requests are retried up to three times after failure.

If requests are likely to pass on retry, enable this property to reduce the overhead of unnecessarily killing and remaking connections.

If requests are likely to fail on retry, disable this property to reduce the retry time.