ASE alerts resolution

The following table describes the various email alerts sent by ASE and their possible resolution. The resolution provided is only a starting point to understand the cause of the alert. If ASE is reporting an alert even after the following the resolution provided, contact PingIntelligence support.

Email alert Possible cause and resolution

Email alert	Possible cause and resolution
ASE start or restart email	When ASE starts or restarts, it sends an email to the configured email ID. If email from ASE is not received, check the email settings in `ase.conf` file.
high CPU usage	Cause: Each ASE node polls for CPU usage of the system every 30-minutes. If the average CPU usage in the 30-minutes interval is higher than the configured threshold in `ase.conf`, then ASE sends an alert. Resolution: If ASE is reporting a high CPU usage, check if other processes are running on the machine on which ASE is installed. If ASE controller or balancer processes are consuming high CPU, it may mean that ASE is receiving high traffic. You should consider adding more ASE nodes.
high memory usage	Cause: Each ASE node polls for memory usage of the system every 30-minutes. If the average memory usage in the 30-minutes interval is higher than the configured threshold `ase.conf`, then ASE sends an alert. Resolution: If ASE is reporting a high memory usage, check if any other process is consuming memory of the system on which ASE is installed. Kill any unnecessary process other than ASE’s process.
high filesystem usage	Cause: Each ASE node polls for filesystem usage of the system every 30-minutes. If the average filesystem usage in the 30-minutes interval is higher than the configured threshold `ase.conf`, then ASE sends an alert. Resolution: If ASE is reporting a high filesystem usage, check if the filesystem is getting full. Run the purge script available in the `util` directory to clear the log files.
API added	ASE sends an email alert when an API is added to ASE using CLI or REST API. Confirm: ASE admin should verify whether correct APIs were added manually or the APIs were added by AAD because of auto-discovery in ABS. If an API is accidentally added, you should immediately remove it from ASE.
API removed	ASE sends an email alert when an API is removed using CLI or REST API. Confirm: ASE admin should verify whether the APIs were deleted intentionally or accidentally.
API updated	ASE sends an email alert when an API definition (the API JSON file) is updated by using CLI or REST API. Confirm: ASE admin should verify whether the correct APIs was updated.
Server added	ASE sends an email alert when a server is added to an API by using CLI or REST API. Confirm: ASE admin should verify whether the correct server was added to API.
Server removed	ASE sends an email alert when a server is removed from an API by using CLI or REST API. Confirm: ASE admin should verify whether the correct server was removed from an API.
cluster node up	ASE sends an email alert when a node joins an ASE cluster. Confirm: ASE admin should verify whether the correct ASE node joined the ASE cluster.
cluster node down	ASE sends an email alert when a node is removed from an ASE cluster. Confirm: ASE admin should check the reason for removal of ASE node from the cluster. ASE node could disconnect from cluster because of network issues, a manual stop of ASE, or change in IP address of the ASE machine.
server state changed to Up	ASE sends an email alert when the backend API server changes state from inactive to active. This alert is applicable for Inline ASE when health check is enabled for an API. This is an informative alert.
server changed to Down	ASE sends an email alert when the backend API server changes state from active to inactive. This alert is applicable for Inline ASE when health check is enabled for an API. Resolution: ASE admin should investigate the reason for the backend API server being not reachable from ASE. You can run the ASE `health_status` command to check the error which caused the server to become inactive.
decoy API accessed	ASE sends an email alert when a decoy API is accessed. This is an informative alert.

ASE start or restart email

When ASE starts or restarts, it sends an email to the configured email ID. If email from ASE is not received, check the email settings in ase.conf file.

high CPU usage

Cause: Each ASE node polls for CPU usage of the system every 30-minutes. If the average CPU usage in the 30-minutes interval is higher than the configured threshold in ase.conf, then ASE sends an alert.

Resolution: If ASE is reporting a high CPU usage, check if other processes are running on the machine on which ASE is installed. If ASE controller or balancer processes are consuming high CPU, it may mean that ASE is receiving high traffic. You should consider adding more ASE nodes.

high memory usage

Cause: Each ASE node polls for memory usage of the system every 30-minutes. If the average memory usage in the 30-minutes interval is higher than the configured threshold ase.conf, then ASE sends an alert.

Resolution: If ASE is reporting a high memory usage, check if any other process is consuming memory of the system on which ASE is installed. Kill any unnecessary process other than ASE’s process.

high filesystem usage

Cause: Each ASE node polls for filesystem usage of the system every 30-minutes. If the average filesystem usage in the 30-minutes interval is higher than the configured threshold ase.conf, then ASE sends an alert.

Resolution: If ASE is reporting a high filesystem usage, check if the filesystem is getting full. Run the purge script available in the util directory to clear the log files.

API added

ASE sends an email alert when an API is added to ASE using CLI or REST API.

Confirm: ASE admin should verify whether correct APIs were added manually or the APIs were added by AAD because of auto-discovery in ABS. If an API is accidentally added, you should immediately remove it from ASE.

API removed

ASE sends an email alert when an API is removed using CLI or REST API.

Confirm: ASE admin should verify whether the APIs were deleted intentionally or accidentally.

API updated

ASE sends an email alert when an API definition (the API JSON file) is updated by using CLI or REST API.

Confirm: ASE admin should verify whether the correct APIs was updated.

Server added

ASE sends an email alert when a server is added to an API by using CLI or REST API.

Confirm: ASE admin should verify whether the correct server was added to API.

Server removed

ASE sends an email alert when a server is removed from an API by using CLI or REST API.

Confirm: ASE admin should verify whether the correct server was removed from an API.

cluster node up

ASE sends an email alert when a node joins an ASE cluster.

Confirm: ASE admin should verify whether the correct ASE node joined the ASE cluster.

cluster node down

ASE sends an email alert when a node is removed from an ASE cluster.

Confirm: ASE admin should check the reason for removal of ASE node from the cluster. ASE node could disconnect from cluster because of network issues, a manual stop of ASE, or change in IP address of the ASE machine.

server state changed to Up

ASE sends an email alert when the backend API server changes state from inactive to active. This alert is applicable for Inline ASE when health check is enabled for an API. This is an informative alert.

server changed to Down

ASE sends an email alert when the backend API server changes state from active to inactive. This alert is applicable for Inline ASE when health check is enabled for an API.

Resolution: ASE admin should investigate the reason for the backend API server being not reachable from ASE. You can run the ASE health_status command to check the error which caused the server to become inactive.

decoy API accessed

ASE sends an email alert when a decoy API is accessed. This is an informative alert.

Alerts for uploading access log files to ABS

ASE sends one or more alerts when it is not able to send access log files to ABS. The following table lists the alerts and possible resolution for the alerts.

Email alert

Possible cause and resolution

Network error

Cause: ABS IP may not be reachable or ASE is not able to connect ABS IP and port.

Resolution:

If there is a firewall in the deployment, check whether firewall is blocking access to ABS.
Check whether ABS is running.
Check whether correct IP address is provided in the abs.conf file.

ABS seed node resolve error

Cause: The hostname provided in abs.conf could not be resolved.

Resolution: Check whether correct IP address is provided in abs.conf file.

ABS SSL handshake error

Cause: SSL handshake error could be because of an invalid CA certificate.

Resolution: Check whether a valid CA certificate is configured in ASE.

ABS authentication error

Cause: Authentication error could be because of invalid access and secret key.

Resolution: Confirm the access key and secret key configured is the same that is configured in ABS abs.properties file.

ABS cluster info error

Cause: Error while fetching ABS cluster information.

Resolution: Check the controller.log file.

ABS config post error

Cause: Error while sending API JSON definition to ABS

Resolution: Check the controller.log file.

ABS service unavailable error

Cause: ABS returning 503 response code.

Resolution: Check the abs.log file.

Log upload error

Cause: API call to upload access log files to ABS fails.

Resolution: Check both ASE’s controller.log and ABS abs.log file.

Duplicate log upload error

This is an informative message.

ABS node queue full error

Cause: ABS responds with a message that it’s queue is full. This can be because of increased traffic on ASE and large number of access log files being generated.

Resolution: Increase the number of ABS nodes.

ABS node capacity low error

Cause: ABS resources are utilized to a maximum.

Resolution: Increase the number of ABS nodes.

ABS attack get error

Cause: Error while fetching attack list from ABS

Resolution: Check ASE’s controller.log file.