Java Agents 2024.11

Troubleshoot

Ping Identity provides support services, professional services, training, and partner services to help you set up and maintain your deployments. Learn more in Getting support.

Get information about the problem

When you are trying to solve a problem, save time by asking the following questions:

  • How do you reproduce the problem?

  • What behavior do you expect, and what behavior do you see?

  • When did the problem start occurring?

  • Are there circumstances in which the problem does not occur?

  • Is the problem permanent, intermittent, getting better, getting worse, or staying the same?

If you contact us for help, include the following information with your request:

  • Description of the problem, including when the problem occurs and its impact on your operation.

  • The product version and build information.

  • Steps you took to reproduce the problem.

  • Relevant access and error logs, stack traces, and core dumps.

  • Description of the environment, including the following information:

    • Machine type

    • Operating system and version

    • Web server or container and version

    • Java version

    • Patches or other software that might affect the problem

WebSocket issues

If you’re experiencing issues with WebSocket connections, perform the following troubleshooting steps:

Check the WebSocket jars are loading

You can check the jars are being loaded on the AM Tomcat as follows:

  1. Run the noisy command from the Tomcat bin directory, for example:

    $ cd /path/to/tomcat/bin
$ lsof | grep websocket

    • If the WebSocket jars are loading, you’ll see responses similar to the following:

      java 1014 root mem REG 8,1 225632 2097375 /usr/local/tomcat/lib/tomcat-websocket.jar
java 1014 root mem REG 8,1 36905 2097376 /usr/local/tomcat/lib/websocket-api.jar
java 1014 root 38r REG 8,1 36905 2097376 /usr/local/tomcat/lib/websocket-api.jar
...

    • If the WebSocket jars aren’t loading, you won’t see them listed.

  2. Add the following option to the Tomcat startup script (setenv.sh) to view further details about the Java WebSocket API loading:

    JAVA_OPTS=-verbose:class

    The Java WebSocket API is bundled with Tomcat.

  3. Restart the web container.

  4. Review the catalina.out log file for WebSocket details. For example, you’ll see entries similar to the following if the WebSocket API is available:

    $ cat ../logs/catalina.out | grep websocket

[Loaded org.forgerock.openam.notifications.websocket.JsonValueDecoder from file:/path/to/tomcat/webapps/am/WEB-INF/lib/openam-notifications-websocket-7.5.0.jar]

[Loaded org.forgerock.openam.notifications.websocket.NotificationsWebSocketConfigurator from file:/path/to/tomcat/webapps/am/WEB-INF/lib/openam-notifications-websocket-7.5.0.jar]

[Loaded javax.websocket.EncodeException from file:/path/to/tomcat/lib/websocket-api.jar]

...
The openam-notifications-websocket-x.x.x.jar is required for WebSockets to work. If it’s missing, you’ll see 404 responses. To resolve this, verify your Tomcat configuration or contact your System Administrator for further assistance.

Test the WebSocket connection

You can test the WebSocket connection by sending the agent’s token to the notifications endpoint using curl.

  1. Authenticate as the agent to return the agent’s token:

    $ curl \
--request POST \
--header "X-OpenAM-Username: agent-id" \ (1)
--header "X-OpenAM-Password: password" \ (2)
--header "Content-Type: application/json" \
--header "Accept-API-Version: resource=2.1" \
'https://am.example.com:8443/am/json/realms/root/realms/alpha/authenticate?auth-service' (3)
    1 Replace agent-id with the ID of the agent profile you created.
    2 Replace password with the agent password.
    3 Replace auth-service with either authIndexType=module&authIndexValue=Application or authIndexType=service&authIndexValue=Agent depending on whether you authenticate using the default non-configurable authentication module or a journey called Agent.

    If authentication is successful, the response includes the tokenId that corresponds to the agent session and the URL to which the agent would normally be redirected. For example:

    {
     "tokenId":"AQIC5wM...​TU3OQ*",
     "successUrl":"/am/console",
     "realm":"/alpha"
}

  2. Send the agent’s token to the notifications endpoint, for example:

    $ curl \
--verbose \
--show-headers \
--no-buffer \
--header "Connection: Upgrade" \
--header "Upgrade: websocket" \
--header "Host: agent.example.com:443" \
--header "Origin: https://notagent.example.com:8443" \
--header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
--header "Sec-WebSocket-Version: 13" \
--header "iPlanetDirectoryPro: AQIC5wM…​​TU3OQ*"
'https://am.example.com:8443/am/notifications'

    The notifications endpoint applies some login processing logic using a servlet filter and returns one of the following responses:

    101 response

    A 101 response indicates everything is ok. It confirms the request is valid, it has been upgraded successfully, and the WebSockets connection is working correctly.

    Example 101 response 
    * Trying 198.51.100.0...
* TCP_NODELAY set
* Connected to am.example.com (198.51.100.0) port 8443 (#0)
> GET /am/notifications HTTP/1.1
> Host: agent.example.com:443
> User-Agent: curl/8.71.0
> Accept: */*
> Connection: Upgrade
> Upgrade: websocket
> Origin: https://notagent.example.com:8443
> Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==
> Sec-WebSocket-Version: 13
> iPlanetDirectoryPro: AQIC5wM…​​TU3OQ*

>

< HTTP/1.1 101
HTTP/1.1 101
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Upgrade: websocket
Upgrade: websocket
< Connection: upgrade
Connection: upgrade
< Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
< Date: Mon, 21 Oct 2024 17:38:15 GMT
Date: Mon, 21 Oct 2024 17:38:15 GMT
    403 response

    A 403 Access Forbidden response means the agent has failed to establish a WebSocket connection with AM. You’ll see 401 responses for this issue in the logs.

    404 response

    A 404 response typically means the WebSocket request has not been upgraded but the request sent to the notifications endpoint is valid. Possible causes for a 404 response are:

    • A network issue such as incorrectly configured load balancers or reverse proxies.

    • A Tomcat issue such as a missing openam-notifications-websocket-x.x.x.jar.

Common issues and solutions

Installation and upgrade

Cannot install over HTTPS
Question

What should I do if I get an error like this when installing an agent with an HTTPS connection to AM:

AM server URL: https://am.example.com:8443/am

WARNING: Unable to connect to AM server URL....

If the AM server is TLS-enabled and the root CA certificate for the AM
server certificate has been not imported into the installer JVMs key store (see
installer-logs/debug/Agent.log for detailed exception), import the root
CA certificate and restart the installer; or continue installation without
verifying the AM server URL.
Answer

The Java platform includes certificates from many certificate authorities (CAs). If, however, you run your own CA, or you use self-signed certificates for HTTPS on the web application container where you run AM, then the agentadmin command cannot trust the certificate presented during connection to AM, and so cannot complete installation correctly.

After setting up the web application container where you run AM to use HTTPS, get the certificate to trust in a certificate file. The certificate you want is that of the CA who signed the container certificate, or the certificate itself if the container certificate is self-signed.

Copy the certificate file to the system where you plan to install the Java agent. Import the certificate into a trust store that you will use during Java agent installation. If you import the certificate into the default trust store for the Java platform, then the agentadmin command can recognize it without additional configuration.

You can find details about export and import of self-signed certificates in Configuring AM’s container for HTTPS in AM’s Installation guide.

Redirection URI error after upgrade
Question

I have upgraded my agent and I see the following message in the agent log:

redirect_uri_mismatch. The redirection URI provided does not match a pre-registered value.

What should I do?

Answer

Java Agent accept only requests sent to the URL specified by the Agent Root URL for CDSSO property. For example, https://agent.example.com/.

As a security measure, agents prevent you from accessing the agent on URLs not defined in the Agent Root URL for CDSSO property. Add entries to this property when:

  • Configuring Alternative Agent Protocol to access the agent through different protocols. For example, http://agent.example.com/ and https://agent.example.com/.

  • Configuring Alternative Agent Host Name to access the agent through different virtual host names. For example, https://agent.example.com/ and https://internal.example.com/.

  • Configuring Alternative Agent Port Number to access the agent through different ports. For example, https://agent.example.com/ and https://agent.example.com:8443/.

Other issues

Detect the path of a resource loaded by classloader
Question

Why does the agent fail to initialize with an error like this:

Servlet failed with an Exception: java.lang.NoSuchMethodError: com.sun.identity.shared.configuration.SystemPropertiesManager.getAsInt(Ljava/lang/String;I)I
Answer

An out-of-date version of the SystemPropertiesManager class is deployed in another jar, that did not have the getAsInt function, and the classloader chose the outdated class instead of the agent’s class.

Follow these steps to help locate jars containing outdated classes by showing where these classes are loaded from:

  1. Add the property -Ddisplay.classpath.mode.enabled=true to the JVM properties in the container where the agent runs, and restart the container.

    JAVA_OPTS="$JAVA_OPTS -Ddisplay.classpath.mode.enabled=true"

  2. Access the URL of a protected resource, using the class name as a query parameter. The following example requests the path of the SystemPropertiesManager package:

    https://www.example.com/myapp/index.html?com.sun.identity.shared.configuration.SystemPropertiesManager

    If the agent finds the package, it displays the path. Otherwise, it displays an error.

  3. After troubleshooting, remove Ddisplay.classpath.mode.enabled=true from the JVM properties. While it is present, the agent returns the class path but performs no other function.

Infinite redirection loops with stateless sessions
Question

I have client-based (stateless) sessions configured in AM, and I am getting infinite redirection loops. In the debug.log file I can see messages similar to the following:

<timestamp> +0000 ERROR [c53...348]state identifier not present in authentication state
<timestamp> +0000 WARNING [c53...348]unable to verify pre-authentication cookie
<timestamp> +0000 WARNING [c53...348]convert_request_after_authn_post(): unable to retrieve pre-authentication request data
<timestamp> +0000 DEBUG [c53...348] exit status: forbidden (3), HTTP status: 403, subrequest 0

What is happening?

Answer

This redirection loop happens because the client-based (stateless) session cookie is surpassing the maximum supported browser header size. Because the cookie is incomplete, AM cannot validate it.

To ensure the session cookie does not surpass the browser supported size, configure either signing and compression or encryption and compression. Learn more in AM’s Security guide.

WebSocket connections
Question

I am seeing errors such as the following:

WARNING: Failed to create new WebSocket connection, backing off
org.forgerock.openam.agents.notifications.websocket.WebSocketConnectionException: Failed to create connection
Answer

Make sure any load balancers or reverse proxies configured in your environment support WebSocket protocols.