Web Agents 2024.9

Troubleshoot

Ping Identity provides support services, professional services, training, and partner services to help you set up and maintain your deployments. Learn more in Getting support.

Get information about the problem

When you are trying to solve a problem, save time by asking the following questions:

  • How do you reproduce the problem?

  • What behavior do you expect, and what behavior do you see?

  • When did the problem start occurring?

  • Are their circumstances in which the problem does not occur?

  • Is the problem permanent, intermittent, getting better, getting worse, or staying the same?

If you contact us for help, include the following information with your request:

  • Description of the problem, including when the problem occurs and its impact on your operation.

  • The product version and build information.

  • Steps you took to reproduce the problem.

  • Relevant access and error logs, stack traces, and core dumps.

  • Description of the environment, including the following information:

    • Machine type

    • Operating system and version

    • Web server and version

    • Java version

    • Patches or other software that might affect the problem

TLS key logging

You can log TLS keys to help troubleshoot and diagnose TLS issues between the agent and AM.

To log TLS keys, you must:

  1. Set the Enable TLS key logging property to true.

  2. Specify the name of the SSL key log file in the AM_SSL_KEYLOG_FILE environment variable.

Only enable TLS key logging when advised by Support. After troubleshooting, disable key logging and remove the SSL key log file.

The SSL key log file contains potentially sensitive TLS transaction data and should be protected from unauthorized access.

Apache Web Agent example

  1. Set the org.forgerock.agents.config.tls.keylog.enable property to true in the agent.conf file.

  2. Set the AM_SSL_KEYLOG_FILE environment variable to a suitable file in the setenv.sh file. The agent must have write access to this file.

  3. Restart the web server.

  4. Start a packet capture using tcpdump on the agent. For example, where AM is listening on port 4443:

    tcpdump -i enp0s8 -s 0 -w /tmp/apache-agent-am.pcap 'port 4443'

  5. Make requests to the agent to initiate traffic from the agent to AM.

  6. Stop the packet capture.

  7. Unset the AM_SSL_KEYLOG_FILE environment variable in the setenv.sh file.

  8. Set the org.forgerock.agents.config.tls.keylog.enable property to false in the agent.conf file.

  9. Restart the web server.

  10. Send the SSL key log file and packet capture to Support for troubleshooting.

  11. Remove the SSL key log file from your system.

Common issues and solutions

The agentadmin command offers a validation mode for the agent that can help you troubleshoot issues in your environment; for example, after an agent upgrade or a network change. Learn more from the --V[i] option in agentadmin command.
Shared memory errors during upgrade or installation
Question

During upgrade or installation, what should I do if I get shared memory errors?

Answer

  1. Stop the web server where the agent is installed.

  2. Delete the following shared memory files:

    • /dev/shm/am_cache_0

    • /dev/shm/am_log_data_0

      Depending on your configuration, the files can be named differently.

  3. Start the agent.

HTTP headers are ignored
Question

When I map a response or attribute to an HTTP header, using the following properties, why is the header ignored:

Answer

When injecting information into HTTP headers, do not use underscores (_) in the header name. Underscores are incompatible with systems that run CGI scripts, and the header can be silently dropped.

Error installing Agents on Windows
Question

I am trying to install a web agent on Windows, which will connect to an AM server running over HTTPS, but the installer reports the following error:

init_ssl(): ssleay32.dll is not available (error: 87)
init_ssl(): libeay32.dll is not available (error: 87)
Answer

If OpenSSL is correctly installed, on Windows 7 or Windows Server 2008 R2 systems, apply the update provided in Microsoft knowledge base article KB2533623. See Microsoft Security Advisory: Insecure library loading could allow remote code execution.

Error installing Agents With SELinux
Question

I am trying to install Web Agent on a server with SELinux enabled in enforcing mode and I am getting error messages after installation, or the web server does not start up. What happened?

Answer

When installing Web Agent on Linux or Unix servers, you must ensure that the user that runs the web server process has read and write permissions for the agent installation directory and files.

If SELinux is enabled in enforcing mode, you must also ensure that SELinux is configured to allow the web server process to perform read and write operations to the agent installation directory and files. By default, SELinux only allows the web server process to read files in well-known authorized locations, such as the /var/www/html directory.

For environments where security can be more relaxed, consider setting SELinux or the httpd_t context in permissive mode for troubleshooting purposes.

You can find details about configuring SELinux in the Linux documentation.

Logs Not Written
Question

Why are logs not being written to /log/system_0.log and /log/monitor_0.pipe files? I am seeing this error:

unable to open event channel
Answer

It is likely that the agent does not have permission to be able to write to the /log/system_0.log and /log/monitor_0.pipe log files.

This can occur if you used the agentadmin --V[i] validator command using a user account that is different to the account used to run your web server.

Run the validator command as the same user that runs the web server, for example, by using the sudo command.

To fix the issue, change the ownership of these files to match the user or group that is running your web server.

Port 80 Used as Default
Question

My Apache HTTP server is not using port 80. When I install Web Agent it defaults to port 80. How do I fix this?

Answer

You probably set ServerName in the Apache HTTP Server configuration to the host name, but did not specify the port number.

Instead, set both the host name and port number for ServerName in the configuration. For example, if you have Apache HTTP Server configured to listen on port 8080, then set ServerName appropriately as in the following excerpt:

<VirtualHost *:8080>
ServerName www.localhost.example:8080
Cannot rotate logs
Question

My web server and Web Agent are installed as root, and the agent cannot rotate logs. I am seeing this error:

Could not rotate log file ... (error: 13)

What should I do?

Answer

If the web server is running with a non-root user, for example, the daemon user, you must ensure that user has the following permissions:

  • Read Permission:

    • /web_agents/agent_name/lib

  • Read and Write Permission:

    • /web_agents/agent_name/instances/agent_nnn

    • /web_agents/agent_name/log

Apply execute permissions on the folders listed above, recursively, for the user that runs the web server.

For IIS or ISAPI agents, change the ownership of the files using the agentadmin --o command. Learn more from agentadmin command.

You may also see similar issues if SELinux is enabled in enforcing mode, and it isn’t configured to allow access to agent directories.
Protection Against Phishing Attacks
Question

How do I increase security against possible phishing attacks through open redirect?

Answer

You can specify a list of valid URL resources against which AM validates the goto and gotoOnFail URL using the Valid goto URL Resource service.

AM only redirects a user if the goto and gotoOnFail URL matches any of the resources specified in this setting. If no setting is present, it is assumed that the goto and gotoOnFail URL is valid.

To set the Valid goto URL Resources, use the AM admin UI, and go to Realms > Realm Name > Services > Add > Validation Service, and then add one or more valid goto URLs.

You can use the "*" wildcard to define resources, where "*" matches all characters except "?". For example, you can use the wildcards, such as https://website.example.com/* or https://website.example.com/*?*. For more specific patterns, use resource names with wildcards as described in Configuring success and failure redirection URLs.

Apache Agent Start Up
Question

I have installed the Unix Apache Web Agent, and neither Apache HTTP Server nor the agent start up or log any message. If I remove the agent, the Apache HTTP Server starts again. What can be the problem?

Answer

To troubleshoot Web Agent or a web server that does not start, set the agent logging level to the maximum by performing the following steps:

  1. Set the environment variable AM_SYSTEM_LOG_LEVEL to All in your command line session. For example:

    $ export AM_SYSTEM_LOG_LEVEL=ALL

  2. Restart the Apache HTTP Server.

  3. Check the logs generated in the /path/to/web_agents/agent_type/log/system_n.log.

    Web Agent reserves memory for the policy and session cache based on the AM_MAX_SESSION_CACHE_SIZE environment variable. If the server where the agent is installed does not have enough shared memory available, the web agent may log messages like the following:

    017-11-10 12:06:00.492 +0000   DEBUG [1:7521][source/shared.c:1451]am_shm_create2() about to create block-clusters_0, size 1074008064
2017-11-10 12:06:00.492 +0000   ERROR [1:7521]am_shm_create2(): ftruncate failed, error: 28

    The error message means the web agent tries to reserve 1074008064 bytes of memory, but there is not enough shared memory available. Several reasons may explain why the shared memory is running low, such as:

    • A new application or additional workload may be stretching the server resources to the limit.

      In this case, ensure that the server has enough shared memory available to satisfy the need of all the applications.

    • A web agent may not have been able to release its shared memory after stopping. Therefore, even if the shared memory is technically not in use, it is still reserved and cannot be reassigned unless freed.

      Different operating systems manage the shared memory in different ways. Refer to your operating system documentation for information about checking shared memory usage.

      You can reduce the amount of memory the web agent reserves for the session and policy cache by setting the AM_MAX_SESSION_CACHE_SIZE environment variable to a value between 1048576 (1 MB) and 1074008064 bytes (1 GB). Learn more from Environment variables.

      Troubleshooting a component that does not start and does not generate logs may be difficult to diagnose. Contact the ForgeRock Support team for more help and information.

Infinite Redirection Loops
Question

I have client-based (stateless) sessions configured in AM, and I am getting infinite redirection loops. In the debug.log file I can see messages similar to the following:

... +0000 ERROR [c5319caa-beeb-5a44-a098-d5575e768348]state identifier not present in authentication state
... +0000 WARNING [c5319caa-beeb-5a44-a098-d5575e768348]unable to verify pre-authentication cookie
... +0000 WARNING [c5319caa-beeb-5a44-a098-d5575e768348]convert_request_after_authn_post(): unable to retrieve pre-authentication request data
... +0000 DEBUG [c5319caa-beeb-5a44-a098-d5575e768348] exit status: forbidden (3), HTTP status: 403, subrequest 0

What is happening?

Answer

The redirection loop happens because the client-based (stateless) session cookie is surpassing the maximum supported browser header size. Since the cookie is incomplete, AM cannot validate it.

To ensure the session cookie does not surpass the browser supported size, configure either signing and compression or encryption and compression.

Learn more from AM’s Security guide.

Errors After Upgrade
Question

I have upgraded my agent and, in the logs, I can see errors similar to the following:

redirect_uri_mismatch. The redirection URI provided does not match a pre-registered value.
com.iplanet.sso.SSOException: Invalid Agent Root URL
com.iplanet.sso.SSOException: Goto URL not valid for the agent Provider ID

What should I do?

Answer

Web Agent accepts only requests sent to the URL specified by the Agent Root URL for CDSSO property. For example, https://agent.example.com:443.

As a security measure, Web Agent prevents you from accessing the agent on URLs not defined in the Agent Root URL for CDSSO property. Add entries to this property when:

  • Accessing the agent through different protocols. For example, http://agent.example.com/ and https://agent.example.com/.

  • Accessing the agent through different virtual host names. For example, https://agent.example.com/ and https://internal.example.com/.

  • Accessing the agent through different ports. For example, https://agent.example.com/ and https://agent.example.com:8443/.

Configuration Not Updated After Upgrade
Question

I have upgraded my Unix Apache or IBM HTTP Server Web Agent, and even though notifications are enabled, the agent does not update its configuration. What is happening?

Answer

Set the web agent logging level to the maximum by performing the following steps:

  1. Set the environment variable AM_SYSTEM_LOG_LEVEL to ALL in your command line session. For example:

    $ export AM_SYSTEM_LOG_LEVEL=ALL

  2. Restart the Apache or IBM HTTP server.

  3. Check the logs generated in the /path/to/web_agents/agent_type/log/system_n.log file.

    Sometimes stopping or upgrading an agent does not clean the pipe file the agent uses to communicate with AM. If the newly started agent cannot create the pipe to communicate with AM because it already exists, the agent would log messages like the following:

    ... UTC   DEBUG [1:10551398][source/monitor.c:503]monitor startup
... UTC   ERROR [102:10551398]monitor unable to get semaphore
... UTC   DEBUG [304:10551398][source/config.c:295]config_initialise():  agent configuration read from cache, agent: / wpa-aix7-Httpd7-32bit

If you see similar error messages, perform the following steps to delete the pipe file:

  1. Stop the Apache or IBM HTTP server.

  2. Change directories to the /tmp directory.

  3. Delete the monitor.pipe file.

  4. Restart the Apache or IBM HTTP server.

Custom Pages Not Displayed
Question

After upgrade, the default Apache welcome page appears instead of my custom error pages. What should I do?

Answer

Check your Apache ErrorDocument configuration. If the custom error pages are not in the document root of the Apache HTTP Server, enclose the ErrorDocument directives in Directory elements. For example:

<Directory "/web/docs">
   ErrorDocument 403 myCustom403Page.html
</Directory>

You can find details about ErrorDocument in the Apache documentation.

Failure After Installation
Question

After starting a web agent installation, I see a failure in the logs:

[../resources/troubleshooting/troubleshooting.bash:#web-agent-install]
Answer

Web Agent installation, can fail if AM’s validation of the agent configuration exceeds the default timeout of 4 seconds.

You can set the AM_NET_TIMEOUT environment variable to change the default timeout, and then rerun the installation.

Agent Not Protecting a Website
Question

My Web Agent is not protecting my website. In the logs, I can see errors similar to the following:

... -0500  ERROR [86169084-5648-6f4d-a706-30f5343d9220]config_fetch():  failed to load configuration for agent: myagent myagent, error -24
... -0500  ERROR [86169084-5648-6f4d-a706-30f5343d9220]amagent_auth_handler(): failed to get agent configuration instance, error: invalid agent session*

What is happening?

Answer

The Web Agent is unable to log in to AM. Possible causes are:

  • Network connection between the agent and AM is unavailable.

  • The AM Connection URL property, which specifies the AM URL may be misconfigured.

Agent Not Protecting a Website
Question

My Web Agent is not protecting my website. In the debug.log file I can see messages similar to the following:

... GMT DEBUG [162ba6eb-cf88-3d7f-f92c-ee8b21971b4c]: (source/oidc.c:265) agent_realm does not have the expected value: JWT
   {
   "sub":"demo",
   "auditTrackingId":"267d1f56-0b97-4830-ae91-6be4b8b7099f-5840",
   "iss":"https://am.example.com:8443/am/oauth2/Customers",
   "tokenName":"id_token",
   "nonce":"D3AE96656D6D634489AF325D90C435A2",
   "aud":"webagent",
   "s_hash":"rxwxIoqDFiwt4MxSwiBa-w",
   "azp":"webagent",
   "auth_time":1561600459,
   "forgerock":{
   "ssotoken":"wi8tHql...MQAA*",
   "suid":"267d1f56-0b97-4830-ae91-6be4b8b7099f-5647"
   },
   "realm":"/Customers",
   "exp":1561607661,
   "tokenType":"JWTToken",
   "iat":1561600461,
   "agent_realm":"/Customers"
   }
... GMT WARNING [162ba6eb-cf88-3d7f-f92c-ee8b21971b4c]: redirect_after_authn(): unable to validate JWT

What is happening?

Answer

If you configured the agent profile in a realm other than AM’s top-level realm (/), you must configure the agent com.sun.identity.agents.config.organization.name bootstrap property with the realm where the agent profile is located. For example, /Customers.

Realm names are case-sensitive. Failure to set the realm name exactly as configured in AM causes the agent to fail to recognize the realm.

Cannot Access Protected Resources

I am getting HTTP 403 Forbidden messages when accessing protected resources, and I can see errors similar to the following in the debug.log file:

... GMT WARNING [69d4632c-82af-b853-0f340vb7b754]: too many pending authentications
... GMT ERROR  [69d4632c-82af-76da-b853-0f340vb7b754]: save_pre_authn_state(): unable to save state for request

What is happening?

Answer

Agents store the progress of authentication with AM in the pre-authentication cookie, agent-authn-tx. This cookie has a maximum size of 4096 bytes, and can fill up if the agent receives many parallel unauthenticated requests to access protected resources.

Learn more from Enable Multivalue for Pre-Authn Cookie.

Cannot Access Protected Resources
Question

I am getting HTTP 403 Forbidden messages when accessing the Web Agent.

Answer

Make sure the Web Agent is executable:

  1. In the terminal where the Web Agent is running, go to /opt/web_agents.

  2. Review and, if necessary, change the permissions for the directory: