PingIntelligence

Email alerts and reports

API Security Enforcer (ASE) sends email notifications as either alerts or reports.

Notification type Description

Alerts

Event-based

Reports

Sent at a configured frequency from 1 - 7 days using email_report

In a cluster deployment, configure the e-mail on the first ASE node. In case the first ASE node is not available, the ASE node with the next highest up-time takes over the task of sending e-mail alerts and daily reports. For more information on ASE cluster, see Administering an ASE cluster.

; Defines report frequency in days [0=no reports, 1=every day, 2=once in two days and max is 7 ; days]
email_report=1
; Specify your email settings
smtp_host=smtp://<smtp-server>
smtp_port=587
; Set this value to true if smtp host support SSL
smtp_ssl=true
; Set this value to true if SSL certificate verification is required
smtp_cert_verification=false
sender_email=
sender_password=
receiver_email=

; Defines threshold for an email alert. For example, if CPU usage is 70%, you will get an
; alert.
cpu_usage=70
memory_usage=70
filesystem_size=70

Email alerts

When you configure alerts, they use the following template:

Event:  <the type of event>
Value:  <the specific trigger for the event>
When:  <the date and time of the event>
Where:  <the IP address or hostname of the server where the event occurred>

The following is an example alert you might receive:

Event : high memory usage
Value : 82.19%
When : 2019-May-16 18:30:00 PST
Where : vortex-132

Email alerts are sent based on the following event categories:

System resource

System resources are polled every 30 minutes to calculate usage. An email alert is sent if the value exceeds the defined threshold.The following system resources are monitored:

CPU

Average CPU usage for a 30 minute interval.

Memory

Memory usage at the 30th minute.

Filesystem

Filesystem usage at the 30th minute.

Configuration

When configuration changes occur, an email alert is sent for these events:

  • Adding or removing an API

  • Adding or deleting a server

  • Nodes of a cluster are UP or DOWN

Decoy API

When decoy APIs are accessed for the first time, an email alert is sent. The time between consecutive alerts is set using decoy_alert_interval in the ase.conf file. The default value is 180 minutes.

For more information on decoy APIs, see In-Context decoy APIs.
ASE-ABS log transfer and communication

ASE sends an alert in the following two conditions:

Access Log transfer failure

When ASE is unable to send access log files to API Behavioral Security (ABS) for more than an hour, ASE sends an alert with the names of the log files.

ASE-ABS communication failure

When interruptions occur in ASE-ABS communication, an alert is sent identifying the error type. The email also mentions the current and total counter for the alert. The current counter lists the number of times that failure happened in the last hour. The total counter lists the total number of times that error has occurred since ASE was started.

  • ABS seed node resolve

  • ABS authentication

  • ABS config post

  • ABS cluster INFO

  • ABS service unavailable

  • Log upload

  • Duplicate log upload

  • Log file read

  • ABS node queue full

  • ABS node capacity low

  • ABS attack type fetch

The following alerts are logged in the controller.log file when email alerts are disabled (enable_email=false) in the ase.conf file:

  • High CPU use

  • High memory use

  • High filesystem use

  • Adding API to ASE

  • Removing API from ASE

  • Updating and API

  • Adding a backend server

  • Removing a backend server

  • ASE cluster node available

  • ASE cluster node unavailable

  • Backend server state changed to UP

  • Backend server state changed to DOWN

  • Log upload service failure

  • Error while uploading file

  • Invalid ASE license file

  • Expired ASE license file

Email reports

ASE sends reports at a frequency in number of days configured in ase.conf file. The report is sent at midnight, 00:00:00 hours based on the local system time.

The report contains the following:

  • Cluster name and location

  • Status information on each cluster node:

    • Operating system, IP address, management port, and cluster port

    • Ports and the number of processes (PIDs)

    • Average CPU memory utilization (average during 30-minute polling intervals)

    • Disk usage and log size

  • Information on each API: Name, Protocol, and Server Pool

The following example shows what a weekly or daily email report looks like:

Date: Sat, 29 Jun 2019 04:01:47 -0800 (PST)
To: receiver@example.com
From: sender@exmple.com
Subject:  API Security Enforcer Daily Reports

Dear DevOps,
Please find the daily report generated by ase2 at 2019-Jun-29 00:01:01 UTC.
============== Cluster Details =================
Cluster Name: pi_cluster
Active Nodes: 2
Inactive nodes: 0
No of APIs: 7
LSM State: disabled
Manual IOC: 0
Automated IOC: 0

================== Node 1 ===================
Host Name: apx1
Management Port: 8010
Cluster Port: 8020
Status: Active
Up Since: 2019-Jan-26 09:27:26
Operating System: Ubuntu 14.04.4 LTS
CPU Usage: 55.80%
Memory Usage: 38.17%
Filesystem Usage: 17.20%
Log Size: 20 GB

================== Node 2 ===================
Host Name : apx2
Management Port: 8010
Cluster Port: 8020
Status: Active
Up Since: 2019-Jan-26 09:26:35
Operating System: Ubuntu 14.04.4 LTS
CPU Usage: 55.79%
Memory Usage: 38.17%
Filesystem Usage: 17.20%
Log Size: 20 GB
=============================================

================= API Details ==================
API ID: https-app
Status: loaded
Protocol: https
decoy: in-context
Active Servers: 172.17.0.8:2800 172.17.0.7:2700
Inactive Servers:
=============================================
API ID: http-app
Status: loaded
Protocol: http
decoy: in-context
Active Servers: 172.17.0.7:2100 172.17.0.8:2300 172.17.0.7:2700
Inactive Servers:
=============================================

Best,
API Security Enforcer

Decoy API access reports

ASE sends decoy API access report at a 3-hour interval by default. You can configure this time interval in minutes in the ase.conf file by configuring the decoy_alert_interval variable. ASE sends the report only if the decoy API is accessed during the configured time interval.

The report provides the following details:

  • The start time when the decoy API was first accessed and the end time when it was last accessed

  • The ASE cluster name

  • The total number of requests for decoy API in the ASE cluster

  • The host name of the ASE where the decoy API was accessed

The following example shows what an email report for a decoy API looks like:

Date: Sat, 29 Jun 2019 04:01:47 -0800 (PST)
To: receiver@example.com
From: sender@exmple.com
Subject:  API Security Enforcer Decoy Access Reports

Dear DevOps,
Please find the decoy report generated by ase2 at 2019-Jun-29 12:01:45 UTC. The default location for the decoy log files is in the directory: /opt/pingidentity/ase/logs/
============== Decoy Summary =================
Cluster Name: pi_cluster
Start Time: 2019-Jun-29 09:00:00
End Time: 2019-Jun-29 12:00:00
Total Requests: 875

================== Node 1 ===================
Host Name: ase2
Total Requests: 428

================== Node 1 ===================
Host Name: ase
Total Requests: 447

Best,
API Security Enforcer

ASE alerts resolution

The following table describes the various email alerts sent by ASE and their possible resolution. The resolution provided is only a starting point to understand the cause of the alert. If ASE is reporting an alert even after the following the resolution provided, contact PingIntelligence for APIs support.

Email alert Possible cause and resolution

ASE start or restart email

When ASE starts or restarts, it sends an email to the configured email ID. If email from ASE is not received, check the email settings in ase.conf file.

High CPU usage

Cause: Each ASE node polls for CPU usage of the system every 30-minutes. If the average CPU usage in the 30-minutes interval is higher than the configured threshold in ase.conf, then ASE sends an alert.

Resolution: If ASE is reporting a high CPU usage, check if other processes are running on the machine on which ASE is installed. If ASE controller or balancer processes are consuming high CPU, it may mean that ASE is receiving high traffic. You should consider adding more ASE nodes.

High memory usage

Cause: Each ASE node polls for memory usage of the system every 30-minutes. If the average memory usage in the 30-minutes interval is higher than the configured threshold ase.conf, then ASE sends an alert.

Resolution: If ASE is reporting a high memory usage, check if any other process is consuming memory of the system on which ASE is installed. Kill any unnecessary process other than ASE’s process.

High filesystem usage

Cause: Each ASE node polls for filesystem usage of the system every 30-minutes. If the average filesystem usage in the 30-minutes interval is higher than the configured threshold ase.conf, then ASE sends an alert.

Resolution: If ASE is reporting a high filesystem usage, check if the filesystem is getting full. Run the purge script available in the util directory to clear the log files.

API added

ASE sends an email alert when an API is added to ASE using CLI or REST API.

Confirm: ASE admin should verify whether correct APIs were added manually or the APIs were added by AAD because of auto-discovery in API Behavioral Security (ABS). If an API is accidentally added, you should immediately remove it from ASE.

API removed

ASE sends an email alert when an API is removed using CLI or REST API.

Confirm: ASE admin should verify whether the APIs were deleted intentionally or accidentally.

API updated

ASE sends an email alert when an API definition (the API JSON file) is updated by using CLI or REST API.

Confirm: ASE admin should verify whether the correct APIs was updated.

Server added

ASE sends an email alert when a server is added to an API by using CLI or REST API.

Confirm: ASE admin should verify whether the correct server was added to API.

Server removed

ASE sends an email alert when a server is removed from an API by using CLI or REST API.

Confirm: ASE admin should verify whether the correct server was removed from an API.

Cluster node up

ASE sends an email alert when a node joins an ASE cluster.

Confirm: ASE admin should verify whether the correct ASE node joined the ASE cluster.

Cluster node down

ASE sends an email alert when a node is removed from an ASE cluster.

Confirm: ASE admin should check the reason for removal of ASE node from the cluster. ASE node could disconnect from cluster because of network issues, a manual stop of ASE, or change in IP address of the ASE machine.

Server state changed to Up

ASE sends an email alert when the backend API server changes state from inactive to active. This alert is applicable for Inline ASE when health check is enabled for an API. This is an informative alert.

Server changed to Down

ASE sends an email alert when the backend API server changes state from active to inactive. This alert is applicable for Inline ASE when health check is enabled for an API.

Resolution: ASE admin should investigate the reason for the backend API server being not reachable from ASE. You can run the ASE health_status command to check the error which caused the server to become inactive.

Decoy API accessed

ASE sends an email alert when a decoy API is accessed. This is an informative alert.

Alerts for uploading access log files to ABS

ASE sends one or more alerts when it is not able to send access log files to ABS. The following table lists the alerts and possible resolution for the alerts.

Email alert Possible cause and resolution

Network error

Cause: ABS IP may not be reachable or ASE is not able to connect ABS IP and port.

Resolution:

  • If there is a firewall in the deployment, check whether firewall is blocking access to ABS.

  • Check whether ABS is running.

  • Check whether correct IP address is provided in the abs.conf file.

ABS seed node resolve error

Cause: The host name provided in the abs.conf file could not be resolved.

Resolution: Check whether correct IP address is provided in abs.conf file.

ABS SSL handshake error

Cause: SSL handshake error could be because of an invalid CA certificate.

Resolution: Check whether a valid CA certificate is configured in ASE.

ABS authentication error

Cause: Authentication error could be because of invalid access and secret key.

Resolution: Confirm the access key and secret key configured is the same that is configured in ABS abs.properties file.

ABS cluster info error

Cause: Error while fetching ABS cluster information.

Resolution: Check the controller.log file.

ABS config post error

Cause: Error while sending API JSON definition to ABS.

Resolution: Check the controller.log file.

ABS service unavailable error

Cause: ABS returning 503 response code.

Resolution: Check the abs.log file.

Log upload error

Cause: API call to upload access log files to ABS fails.

Resolution: Check both ASE’s controller.log and ABS abs.log file.

Duplicate log upload error

This is an informative message.

ABS node queue full error

Cause: ABS responds with a message that it’s queue is full. This can be because of increased traffic on ASE and large number of access log files being generated.

Resolution: Increase the number of ABS nodes.

ABS node capacity low error

Cause: ABS resources are utilized to a maximum.

Resolution: Increase the number of ABS nodes.

ABS attack get error

Cause: Error while fetching attack list from ABS.

Resolution: Check ASE’s controller.log file.