Web Agents 2024.11

Troubleshoot

Ping Identity provides support services, professional services, training, and partner services to help you set up and maintain your deployments. Learn more in Getting support.

Get information about the problem

When you are trying to solve a problem, save time by asking the following questions:

  • How do you reproduce the problem?

  • What behavior do you expect, and what behavior do you see?

  • When did the problem start occurring?

  • Are there circumstances in which the problem does not occur?

  • Is the problem permanent, intermittent, getting better, getting worse, or staying the same?

If you contact us for help, include the following information with your request:

  • Description of the problem, including when the problem occurs and its impact on your operation.

  • The product version and build information.

  • Steps you took to reproduce the problem.

  • Relevant access and error logs, stack traces, and core dumps.

  • Description of the environment, including the following information:

    • Machine type

    • Operating system and version

    • Web server and version

    • Java version

    • Patches or other software that might affect the problem

WebSocket issues

If you’re experiencing issues with WebSocket connections, perform the following troubleshooting steps:

Validate the agent

The agentadmin --V command performs a number of checks, including WebSocket tests. Learn more in agentadmin --V.

The results of the tests are output to the command line and show tests as ok or not ok depending on whether they passed, for example:

...
validate_system_resources: ok
validate_session_profile: skipped
Agent websocket open error: error (6)
validate_websocket_connection: not ok
...

You can find further information about the tests and detailed results in the Validator log (validate_nn.log located in the agent /log directory).

Check the WebSocket jars are loading

You can check the jars are being loaded on the AM Tomcat as follows:

  1. Run the noisy command from the Tomcat bin directory, for example:

    $ cd /path/to/tomcat/bin
$ lsof | grep websocket

    • If the WebSocket jars are loading, you’ll see responses similar to the following:

      java 1014 root mem REG 8,1 225632 2097375 /usr/local/tomcat/lib/tomcat-websocket.jar
java 1014 root mem REG 8,1 36905 2097376 /usr/local/tomcat/lib/websocket-api.jar
java 1014 root 38r REG 8,1 36905 2097376 /usr/local/tomcat/lib/websocket-api.jar
...

    • If the WebSocket jars aren’t loading, you won’t see them listed.

  2. Add the following option to the Tomcat startup script (setenv.sh) to view further details about the Java WebSocket API loading:

    JAVA_OPTS=-verbose:class

    The Java WebSocket API is bundled with Tomcat.

  3. Restart the web container.

  4. Review the catalina.out log file for WebSocket details. For example, you’ll see entries similar to the following if the WebSocket API is available:

    $ cat ../logs/catalina.out | grep websocket

[Loaded org.forgerock.openam.notifications.websocket.JsonValueDecoder from file:/path/to/tomcat/webapps/am/WEB-INF/lib/openam-notifications-websocket-7.5.0.jar]

[Loaded org.forgerock.openam.notifications.websocket.NotificationsWebSocketConfigurator from file:/path/to/tomcat/webapps/am/WEB-INF/lib/openam-notifications-websocket-7.5.0.jar]

[Loaded javax.websocket.EncodeException from file:/path/to/tomcat/lib/websocket-api.jar]

...
The openam-notifications-websocket-x.x.x.jar is required for WebSockets to work. If it’s missing, you’ll see 404 responses. To resolve this, verify your Tomcat configuration or contact your System Administrator for further assistance.

Test the WebSocket connection

You can test the WebSocket connection by sending the agent’s token to the notifications endpoint using curl. This test generates a response similar to what is output in the Validator log.

  1. Authenticate as the agent to return the agent’s token:

    $ curl \
--request POST \
--header "X-OpenAM-Username: agent-id" \ (1)
--header "X-OpenAM-Password: password" \ (2)
--header "Content-Type: application/json" \
--header "Accept-API-Version: resource=2.1" \
'https://am.example.com:8443/am/json/realms/root/realms/alpha/authenticate?auth-service' (3)
    1 Replace agent-id with the ID of the agent profile you created.
    2 Replace password with the agent password.
    3 Replace auth-service with either authIndexType=module&authIndexValue=Application or authIndexType=service&authIndexValue=Agent depending on whether you authenticate using the default non-configurable authentication module or a journey called Agent.

    If authentication is successful, the response includes the tokenId that corresponds to the agent session and the URL to which the agent would normally be redirected. For example:

    {
     "tokenId":"AQIC5wM...​TU3OQ*",
     "successUrl":"/am/console",
     "realm":"/alpha"
}

  2. Send the agent’s token to the notifications endpoint, for example:

    $ curl \
--verbose \
--show-headers \
--no-buffer \
--header "Connection: Upgrade" \
--header "Upgrade: websocket" \
--header "Host: agent.example.com:443" \
--header "Origin: https://notagent.example.com:8443" \
--header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
--header "Sec-WebSocket-Version: 13" \
--header "iPlanetDirectoryPro: AQIC5wM…​​TU3OQ*"
'https://am.example.com:8443/am/notifications'

    The notifications endpoint applies some login processing logic using a servlet filter and returns one of the following responses:

    101 response

    A 101 response indicates everything is ok. It confirms the request is valid, it has been upgraded successfully, and the WebSockets connection is working correctly.

    Example 101 response 
    * Trying 198.51.100.0...
* TCP_NODELAY set
* Connected to am.example.com (198.51.100.0) port 8443 (#0)
> GET /am/notifications HTTP/1.1
> Host: agent.example.com:443
> User-Agent: curl/8.71.0
> Accept: */*
> Connection: Upgrade
> Upgrade: websocket
> Origin: https://notagent.example.com:8443
> Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==
> Sec-WebSocket-Version: 13
> iPlanetDirectoryPro: AQIC5wM…​​TU3OQ*

>

< HTTP/1.1 101
HTTP/1.1 101
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Upgrade: websocket
Upgrade: websocket
< Connection: upgrade
Connection: upgrade
< Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
< Date: Mon, 21 Oct 2024 17:38:15 GMT
Date: Mon, 21 Oct 2024 17:38:15 GMT
    403 response

    A 403 Access Forbidden response means the agent has failed to establish a WebSocket connection with AM. You’ll see 401 responses for this issue in the logs.

    A common reason for a 403 or 401 response is a mismatch between the agent cookie name and the cookie name in AM. The agent cookie name is used to construct the request sent to the notifications endpoint and must match what AM is expecting. Learn more in Cookies.

    404 response

    A 404 response typically means the WebSocket request has not been upgraded but the request sent to the notifications endpoint is valid. Possible causes for a 404 response are:

    • A network issue such as incorrectly configured load balancers or reverse proxies.

    • A Tomcat issue such as a missing openam-notifications-websocket-x.x.x.jar.

TLS key logging

You can log TLS keys to help troubleshoot and diagnose TLS issues between the agent and AM.

To log TLS keys, you must:

  1. Set the Enable TLS key logging property to true.

  2. Specify the name of the SSL key log file in the AM_SSL_KEYLOG_FILE environment variable.

Only enable TLS key logging when advised by Support. After troubleshooting, disable key logging and remove the SSL key log file.

The SSL key log file contains potentially sensitive TLS transaction data and should be protected from unauthorized access.

Apache Web Agent example

  1. Set the org.forgerock.agents.config.tls.keylog.enable property to true in the agent.conf file.

  2. Set the AM_SSL_KEYLOG_FILE environment variable to a suitable file in the setenv.sh file. The agent must have write access to this file.

  3. Restart the web server.

  4. Start a packet capture using tcpdump on the agent. For example, where AM is listening on port 4443:

    tcpdump -i enp0s8 -s 0 -w /tmp/apache-agent-am.pcap 'port 4443'

  5. Make requests to the agent to initiate traffic from the agent to AM.

  6. Stop the packet capture.

  7. Unset the AM_SSL_KEYLOG_FILE environment variable in the setenv.sh file.

  8. Set the org.forgerock.agents.config.tls.keylog.enable property to false in the agent.conf file.

  9. Restart the web server.

  10. Send the SSL key log file and packet capture to Support for troubleshooting.

  11. Remove the SSL key log file from your system.

Common issues and solutions

The agentadmin command offers a validation mode for the agent that can help you troubleshoot issues in your environment; for example, after an agent upgrade or a network change. Learn more in agentadmin --V.

Installation and upgrade

Shared memory errors during upgrade or installation
Question

During upgrade or installation, what should I do if I get shared memory errors?

Answer

  1. Stop the web server where the agent is installed.

  2. Delete the following shared memory files:

    • /dev/shm/am_cache_0

    • /dev/shm/am_log_data_0

      Depending on your configuration, the files could be named differently.

  3. Start the agent.

Error installing agents with SELinux
Question

I am trying to install Web Agent on a server with SELinux enabled in enforcing mode, and I am getting error messages after installation or the web server does not start up. What happened?

Answer

When installing Web Agent on Linux or Unix servers, you must ensure that the user that runs the web server process has read and write permissions for the agent installation directory and files.

If SELinux is enabled in enforcing mode, you must also ensure that SELinux is configured to allow the web server process to perform read and write operations to the agent installation directory and files. By default, SELinux only allows the web server process to read files in well-known authorized locations, such as the /var/www/html directory.

For environments where security can be more relaxed, consider setting SELinux or the httpd_t context in permissive mode for troubleshooting purposes.

You can find details about configuring SELinux in the Linux documentation.

Failure after installation
Question

After starting a web agent installation, I see a failure in the logs:

[../resources/troubleshooting/troubleshooting.bash:#web-agent-install]
Answer

Web Agent installation, can fail if AM’s validation of the agent configuration exceeds the default timeout of 4 seconds.

You can set the AM_NET_TIMEOUT environment variable to change the default timeout, and then rerun the installation.

Errors after upgrade
Question

I have upgraded my agent and, in the logs, I can see errors similar to the following:

redirect_uri_mismatch. The redirection URI provided does not match a pre-registered value.
com.iplanet.sso.SSOException: Invalid Agent Root URL
com.iplanet.sso.SSOException: Goto URL not valid for the agent Provider ID

What should I do?

Answer

Web Agent accepts only requests sent to the URL specified by the Agent Root URL for CDSSO property. For example, https://agent.example.com:443.

As a security measure, Web Agent prevents you from accessing the agent on URLs not defined in the Agent Root URL for CDSSO property. Add entries to this property when:

  • Accessing the agent through different protocols. For example, http://agent.example.com/ and https://agent.example.com/.

  • Accessing the agent through different virtual host names. For example, https://agent.example.com/ and https://internal.example.com/.

  • Accessing the agent through different ports. For example, https://agent.example.com/ and https://agent.example.com:8443/.

Configuration not updated after upgrade
Question

I have upgraded my Unix Apache or IBM HTTP Server Web Agent, and even though notifications are enabled, the agent does not update its configuration. What is happening?

Answer

Set the web agent logging level to the maximum by performing the following steps:

  1. Set the environment variable AM_SYSTEM_LOG_LEVEL to ALL in your command line session. For example:

    $ export AM_SYSTEM_LOG_LEVEL=ALL

  2. Restart the Apache or IBM HTTP server.

  3. Check the logs generated in the /path/to/web_agents/agent_type/log/system_n.log file.

    Sometimes stopping or upgrading an agent does not clean the pipe file the agent uses to communicate with AM. If the newly started agent cannot create the pipe to communicate with AM because it already exists, the agent would log messages like the following:

    ... UTC   DEBUG [1:10551398][source/monitor.c:503]monitor startup
... UTC   ERROR [102:10551398]monitor unable to get semaphore
... UTC   DEBUG [304:10551398][source/config.c:295]config_initialise():  agent configuration read from cache, agent: / wpa-aix7-Httpd7-32bit

If you see similar error messages, perform the following steps to delete the pipe file:

  1. Stop the Apache or IBM HTTP server.

  2. Change directories to the /tmp directory.

  3. Delete the monitor.pipe file.

  4. Restart the Apache or IBM HTTP server.

Start up

Apache agent start up
Question

I have installed the Unix Apache Web Agent, and neither Apache HTTP Server nor the agent start up or log any message. If I remove the agent, the Apache HTTP Server starts again. What can be the problem?

Answer

To troubleshoot Web Agent or a web server that does not start, set the agent logging level to the maximum by performing the following steps:

  1. Set the environment variable AM_SYSTEM_LOG_LEVEL to All in your command line session. For example:

    $ export AM_SYSTEM_LOG_LEVEL=ALL

  2. Restart the Apache HTTP Server.

  3. Check the logs generated in the /path/to/web_agents/agent_type/log/system_n.log.

    Web Agent reserves memory for the policy and session cache based on the AM_MAX_SESSION_CACHE_SIZE environment variable. If the server where the agent is installed does not have enough shared memory available, the web agent may log messages like the following:

    017-11-10 12:06:00.492 +0000   DEBUG [1:7521][source/shared.c:1451]am_shm_create2() about to create block-clusters_0, size 1074008064
2017-11-10 12:06:00.492 +0000   ERROR [1:7521]am_shm_create2(): ftruncate failed, error: 28

    The error message means the web agent tries to reserve 1074008064 bytes of memory, but there is not enough shared memory available. Several reasons may explain why the shared memory is running low, such as:

    • A new application or additional workload may be stretching the server resources to the limit.

      In this case, ensure that the server has enough shared memory available to satisfy the need of all the applications.

    • A web agent may not have been able to release its shared memory after stopping. Therefore, even if the shared memory is technically not in use, it is still reserved and cannot be reassigned unless freed.

      Different operating systems manage the shared memory in different ways. Refer to your operating system documentation for information about checking shared memory usage.

      You can reduce the amount of memory the web agent reserves for the session and policy cache by setting the AM_MAX_SESSION_CACHE_SIZE environment variable to a value between 1048576 (1 MB) and 1074008064 bytes (1 GB). Learn more in Environment variables.

      Troubleshooting a component that does not start and does not generate logs may be difficult to diagnose. Contact Support for more help and information.

Logs

Logs not written
Question

Why are logs not being written to /log/system_0.log and /log/monitor_0.pipe files? I am seeing this error:

unable to open event channel
Answer

It is likely that the agent does not have permission to be able to write to the /log/system_0.log and /log/monitor_0.pipe log files.

This can occur if you used the agentadmin --V[i] validator command using a user account that is different to the account used to run your web server.

Run the validator command as the same user that runs the web server, for example, by using the sudo command.

To fix the issue, change the ownership of these files to match the user or group that is running your web server.

Cannot rotate logs
Question

My web server and Web Agent are installed as root, and the agent cannot rotate logs. I am seeing this error:

Could not rotate log file ... (error: 13)

What should I do?

Answer

If the web server is running with a non-root user, for example, the daemon user, you must ensure that user has the following permissions:

  • Read Permission:

    • /web_agents/agent_name/lib

  • Read and Write Permission:

    • /web_agents/agent_name/instances/agent_nnn

    • /web_agents/agent_name/log

Apply execute permissions on the folders listed above, recursively, for the user that runs the web server.

For IIS or ISAPI agents, change the ownership of the files using the agentadmin --o command. Learn more in agentadmin command.

You may also see similar issues if SELinux is enabled in enforcing mode, and it isn’t configured to allow access to agent directories.

Other issues

HTTP headers are ignored
Question

When I map a response or attribute to an HTTP header, using the following properties, why is the header ignored:

Answer

When injecting information into HTTP headers, do not use underscores (_) in the header name. Underscores are incompatible with systems that run CGI scripts, and the header can be silently dropped.

Port 80 used as default
Question

My Apache HTTP server is not using port 80. When I install Web Agent it defaults to port 80. How do I fix this?

Answer

You probably set ServerName in the Apache HTTP Server configuration to the host name, but did not specify the port number.

Instead, set both the host name and port number for ServerName in the configuration. For example, if you have Apache HTTP Server configured to listen on port 8080, then set ServerName appropriately as in the following excerpt:

<VirtualHost *:8080>
ServerName www.localhost.example:8080
Protection against phishing attacks
Question

How do I increase security against possible phishing attacks through open redirect?

Answer

You can specify a list of valid URL resources against which AM validates the goto and gotoOnFail URL using the Valid goto URL Resource service.

AM only redirects a user if the goto and gotoOnFail URL matches any of the resources specified in this setting. If no setting is present, it is assumed that the goto and gotoOnFail URL is valid.

To set the Valid goto URL Resources, use the AM admin UI, and go to Realms > Realm Name > Services > Add > Validation Service, and then add one or more valid goto URLs.

You can use the "*" wildcard to define resources, where "*" matches all characters except "?". For example, you can use the wildcards, such as https://website.example.com/* or https://website.example.com/*?*. For more specific patterns, use resource names with wildcards as described in Configuring success and failure redirection URLs.

Infinite redirection loops
Question

I have client-based (stateless) sessions configured in AM, and I am getting infinite redirection loops. In the debug.log file I can see messages similar to the following:

... +0000 ERROR [c5319caa-beeb-5a44-a098-d5575e768348]state identifier not present in authentication state
... +0000 WARNING [c5319caa-beeb-5a44-a098-d5575e768348]unable to verify pre-authentication cookie
... +0000 WARNING [c5319caa-beeb-5a44-a098-d5575e768348]convert_request_after_authn_post(): unable to retrieve pre-authentication request data
... +0000 DEBUG [c5319caa-beeb-5a44-a098-d5575e768348] exit status: forbidden (3), HTTP status: 403, subrequest 0

What is happening?

Answer

The redirection loop happens because the client-based (stateless) session cookie is surpassing the maximum supported browser header size. Since the cookie is incomplete, AM cannot validate it.

To ensure the session cookie does not surpass the browser supported size, configure either signing and compression or encryption and compression.

Learn more in AM’s Security guide.

Custom pages aren’t displayed
Question

After upgrade, the default Apache welcome page appears instead of my custom error pages. What should I do?

Answer

Check your Apache ErrorDocument configuration. If the custom error pages are not in the document root of the Apache HTTP Server, enclose the ErrorDocument directives in Directory elements. For example:

<Directory "/web/docs">
   ErrorDocument 403 myCustom403Page.html
</Directory>

You can find details about ErrorDocument in the Apache documentation.

Agent not protecting a website
Question

My Web Agent is not protecting my website. In the logs, I can see errors similar to the following:

... -0500  ERROR [86169084-5648-6f4d-a706-30f5343d9220]config_fetch():  failed to load configuration for agent: myagent myagent, error -24
... -0500  ERROR [86169084-5648-6f4d-a706-30f5343d9220]amagent_auth_handler(): failed to get agent configuration instance, error: invalid agent session*

What is happening?

Answer

The Web Agent is unable to log in to AM. Possible causes are:

  • Network connection between the agent and AM is unavailable.

  • The AM Connection URL property, which specifies the AM URL may be misconfigured.

Agent not protecting a website
Question

My Web Agent is not protecting my website. In the debug.log file I can see messages similar to the following:

... GMT DEBUG [162ba6eb-cf88-3d7f-f92c-ee8b21971b4c]: (source/oidc.c:265) agent_realm does not have the expected value: JWT
   {
   "sub":"demo",
   "auditTrackingId":"267d1f56-0b97-4830-ae91-6be4b8b7099f-5840",
   "iss":"https://am.example.com:8443/am/oauth2/Customers",
   "tokenName":"id_token",
   "nonce":"D3AE96656D6D634489AF325D90C435A2",
   "aud":"webagent",
   "s_hash":"rxwxIoqDFiwt4MxSwiBa-w",
   "azp":"webagent",
   "auth_time":1561600459,
   "forgerock":{
   "ssotoken":"wi8tHql...MQAA*",
   "suid":"267d1f56-0b97-4830-ae91-6be4b8b7099f-5647"
   },
   "realm":"/Customers",
   "exp":1561607661,
   "tokenType":"JWTToken",
   "iat":1561600461,
   "agent_realm":"/Customers"
   }
... GMT WARNING [162ba6eb-cf88-3d7f-f92c-ee8b21971b4c]: redirect_after_authn(): unable to validate JWT

What is happening?

Answer

If you configured the agent profile in a realm other than AM’s top-level realm (/), you must configure the agent com.sun.identity.agents.config.organization.name bootstrap property with the realm where the agent profile is located. For example, /Customers.

Realm names are case-sensitive. Failure to set the realm name exactly as configured in AM causes the agent to fail to recognize the realm.

Can’t access protected resources
Question

I am getting HTTP 403 Forbidden messages when accessing protected resources, and I can see errors similar to the following in the debug.log file:

... GMT WARNING [69d4632c-82af-b853-0f340vb7b754]: too many pending authentications
... GMT ERROR  [69d4632c-82af-76da-b853-0f340vb7b754]: save_pre_authn_state(): unable to save state for request

What is happening?

Answer

Agents store the progress of authentication with AM in the pre-authentication cookie, agent-authn-tx. This cookie has a maximum size of 4096 bytes, and can fill up if the agent receives many parallel unauthenticated requests to access protected resources.

Learn more in Enable Multivalue for Pre-Authn Cookie.

Can’t access protected resources
Question

I am getting HTTP 403 Forbidden messages when accessing the Web Agent.

Answer

Make sure the Web Agent is executable:

  1. In the terminal where the Web Agent is running, go to /opt/web_agents.

  2. Review and, if necessary, change the permissions for the directory:

WebSocket connections
Question

I am seeing errors such as the following:

WARNING: Failed to create new WebSocket connection, backing off
org.forgerock.openam.agents.notifications.websocket.WebSocketConnectionException: Failed to create connection
Answer

Make sure any load balancers or reverse proxies configured in your environment support WebSocket protocols.