Tune thresholds for false positives

ABS automatically generates attack thresholds which are used by the machine learning system to identify attacks and anomalies. Initial attack thresholds are determined based on training and production traffic in your API ecosystem. At the end of the training period, ABS calculates the first set of system-generated threshold values and uses these values to detect attacks.

By default, system generated threshold values are updated every 24-hours. This frequency can be changed at start-up by modifying attack_update_interval using Global configuration update REST API or anytime by using the update.sh script available in the util directory. The minimum value is 1-hour as sufficient traffic is required to update the model.

You can change the threshold period at anytime by running the update.sh script. The value of the updated threshold period is applicable immediately. For example, if the current threshold update period is 10 hours and the new threshold period is 12 hours, then the AI model updates the threshold at the 12th hour.

Access script help by logging into the MongoDB machine and running the following command:

/opt/pingidentity/mongo/update.sh help

Example: change the training period and threshold interval together

/opt/pingidentity/mongo/update.sh -u absuser -p abs123 --attack_initial_training 24 --attack_update_interval 24
updating attack_initial_training to 24
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
updating attack_update_interval to 24
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
The current values of the variables are:
attack_initial_training=24
attack_update_interval=24
api_discovery=true
discovery_initial_interval=48

Check threshold values

Threshold values can be checked using the ABS Threshold API. For each attack type, one or more variables (for example, Var A, B) is used by the machine learning process during attack detection. All variables have a Normal Threshold Value (tn), and some variables also have an Extreme Threshold Value (tx). These values are used during the attack detection process and automatically update over time to provide improved accuracy.

To view the current threshold settings, use the GET method with the following ABS threshold API:

https://<ip_address>:<port>/v4/abs/attack/threshold?api=<api_name>;

The IP address and port corresponding to the host ABS machine. The API payload returned is a JSON file which shows the threshold values for each attack type. See Get Threshold API for an example.

Change attack thresholds

Ping Identity recommends using the automatically generated system thresholds in your production operations. However, if attacks are detected for legitimate traffic (i.e. false positives), then manual tuning options are provided. An administrator has two choices:

Change the system generated threshold value to a larger user-generated value.
Disable the variable to stop detecting attacks (see Disabling Attacks)

To identify settings to change, generate an attack report, which includes attacks known to be false positives. For each identified attack, an Attack Code (for example, "varA (Tn), varB (Tn)") is listed with the threshold variable(s) that triggered the attack. The Attack Code includes the responsible variables (for example, A, B) and threshold types (for example, Tn, Tx); the threshold type can be manually adjusted. Ping Identity recommends slowly increasing the triggered threshold value(s) using user-generated thresholds. After each update, evaluate the new setting to see if false positives are reduced. The process can be repeated until the issue is addressed.

The Threshold API PUT method is used to manually override the system generated setting with a user-defined value. When configuring the threshold manually, the normal threshold (tn), the extreme threshold (tx), or either threshold can be individually set.

You can also use Attack management in Dashboard to tune threshold values for a specific client identifier.

Make sure that you are in Manually set thresholds mode before changing the threshold manually.

Change threshold value Tn only

The Tn threshold value can be changed for each attack type for a specific API. The initial Tx value is automatically calculated based on the gap between the values of Tn and Tx. This gap is determined at the end of the training period. The minimum gap is 1, and the value of Tx always bigger than Tn. Here is an example:

Values at end of training period:

Tn = 12
Tx = 16
Gap = 4 (Tx-Tn)

Threshold API is used to set Tn=13for an API variable.

Tx = 17 (Gap value of 4 is automatically added to new Tnvalue)

This difference between the value of Tn and Tx is maintained when only Tn is moved. However, the difference between the value of Tn and Tx can be changed when only Tx is changed.

The value of Tn can never be more than the value of Tx.

Changing Threshold Value Tx only

Change the Tx value to adjust the gap between the normal and extreme threshold setting for an attack type on a specific API. The value of Tx defines the gap which ranges from a minimum of 1 to the maximum value defined in Threshold range for Tn and Tx. When Tx is moved, the system calculated gap calculated at the end of the training period is no longer used. For the attack types where Tx is not applicable to the variable, “na” is displayed in the threshold API.

If the value of only Tn is moved without modifying Tx, then the new gap between the value of Tn and Tx is used until the value of Tx is changed again.

Change threshold value Tn and Tx together

Both Tn and Tx can be changed for an attack type on a specific API. When Tn and Tx are moved simultaneously, the newly defined value of Tn and gap for Tx are changed. The ranges of Tn and Tx values are detailed in Threshold range for Tn and Tx.

How to configure threshold value

To manually set a threshold, use the PUT method with the following ABS attack API:

https://<ip_address>:<port>/v5/abs/attack/threshold?api=<api_name>;

The IP address and port correspond to the host ABS machine. The API input payload is a JSON file which sets the threshold value for attack types. The parameters include attack type and Normal Threshold (tn) value. When manually setting the threshold for a variable, ABS Threshold API displays both system generated and user configured threshold values. ABS applies the user configured threshold values until it is reconfigured to use system generated values (see below).

Manually set thresholds

The threshold API with PUT method sets the operation mode for the variable by configuring mode to system or user. The following snippet of Threshold API with PUT method shows how to change the threshold mode from system to user and change value of tn, tx, or both at the same time. If you do not wish to change the value for tn or tx in user mode, leave the field blank by putting “” in the Threshold API body. In the following snippet, the value of tn and tx both are changed.

{
 "api_name" : "atmapp",
  "mode": "user",
 "ioc_threshold": [
 {
 "type": "api_memory_attack_type_2",
 "variable": "A",
 "tn": "9",
 "tx": "12"
 },
 {
 "type": "data_exfiltration_attack",
 "variable": "A",
 "tn": "18",
 "tx": ""
 },
 {
 "type": "data_exfiltration_attack",
 "variable": "B",
 "tn": "18",
 "tx": ""
 },
 {
 "type": "api_memory_attack_type_1",
 "variable": "A",
 "tn": "18",
 "tx": ""
 }
 ]
}
{
 "api_name" : "shop",
 "mode": "user",
 "ioc_threshold": [
 {
 "type": "api_memory_attack_type_2",
 "variable": "A",
 "tn": "13"
 },
 {
 "type": "api_memory_attack_type_2",
 "variable": "B",
 "tn": "10"
 }
}

The API response is displayed below:

{
 "message": success: "Thresholds set to user mode for given variables.",
 "date": "Mon Jan 08 15:36:05 IST 2018"
}

After a threshold value is manually set, ABS uses the updated user threshold values to detect attacks.

When threshold mode is changed back to system, the user-configured values are no longer used or displayed in the threshold API output. The following snippet shows changing threshold to system mode from user mode for two variables associated with an API memory attack:

{
 "api_name" : "shop",
  "mode": "system",
 "ioc_threshold": [
 {
 "type": "api_memory_attack_type_2",
 "variable": "A",
 },
 {
 "type": "api_memory_attack_type_2",
 "variable": "B",
 }
}