PingIntelligence

Purge MongoDB data

The API Behavioral Security (ABS) MongoDB purge script dumps and/or deletes processed AI Engine and machine learning data from MongoDB.

It is recommended to archive the data before purging it. The purge_mongo.sh script is available in the/<pi-install-dir>/pingidentity/abs/util directory.

The script offers three options:

  • Only purge data

  • Only dump data

  • Dump data into a specified directory and then purge it

Prerequisites-Ensure that the following prerequisites are fulfilled, before using the script:

  • Execute the script from an ABS AI Engine node with connectivity to the MongoDB primary node.

  • The necessary write permissions are available on the directory where the data dump is stored.

  • Database names used as command line arguments like the data_dbname and mldata_dbname should be same as configured in ABS configuration - abs.properties file.

It is recommended to execute the script during low load periods or during ABS down times.

Thepurge_mongo.sh script supports the following arguments:

Argument Description

--data_db <abs database>

Use this argument to specify the name of the ABS database.

--mldata_db <ml database>

Use this argument to specify the name of the ML database.

You must specify at least one database while executing the script. You can also specify both the databases in a single command.

-d <days> or --days <days>

Number of days of data to be retained. The default number of days is seven. The minimum number of days that can be specified is one and the maximum is 365.

-l <path_to_dump_dir> or --location < path_to_dump_dir>

The directory path to store the MongoDB dump.

--purge_only

Use this option when you only need to delete the data and not take a data dump.

--dump_only

Use this option when you need to take the data dump without deleting the data.

--h or --help

Use this argument for more information on the purge script parameters.

--gzip

Use this argument to compress the data dump. It can be used with dump only and purge and dump options. This argument cannot be used with purge only option.

The following sections show the sample usage ofpurge_mongo.sh script with the three options:

Purge data

The following are a few sample commands to purge the MongoDB data:

  • ./purge_mongo.sh --data_db <abs database> -d <days> --purge_only

  • ./purge_mongo.sh --mldata_db <ml database> -d <days> --purge_only

  • ./purge_mongo.sh --data_db <abs_database> --mldata_db <ml database>d <days> --purge_only

For example, the following command deletes ABS data older than 80 days.

./purge_mongo.sh --data_db abs_data -d 80 --purge_only
Starting the purge mongo tool
This will delete the documents in  abs_data database that are older than 80 days.
Are you sure (yes/no): yes
Deleting the documents in  abs_data database that are older than 80 days.
Please see /opt/pingidentity/abs/logs/purge/purge.log.2020-11-16-05-01-42 for more details

Dump data

The following are a few sample commands to purge the MongoDB data:

  • ./purge_mongo.sh --data_db <abs database> -d <days> -l <path> --dump_only

  • ./purge_mongo.sh --mldata_db <ml database> -d <days> -l <path> --dump_only

  • ./purge_mongo.sh --data_db <abs_database> --mldata_db <ml database>d <days> -l <path> --dump_only

For example, the following command dumps data older than 80 days from ml_database into a /tmp directory. It does not delete the data. /tmp is used as an example reference here, you can substitute /tmp with any other directory path in your environment.

./purge_mongo.sh --mldata_db abs_mldata -d 80 -l /tmp --dump_only
Starting the purge mongo tool
Storing abs ml data from mongo at /tmp/ml_mongo_data.2020-11-16-06-09-06
Please see /opt/pingidentity/abs/logs/purge/purge.log.2020-11-16-06-10-42 for more details

Purge and dump data

The following are a few sample commands to purge and dump the data:

  • ./purge_mongo.sh --data_db <abs database> -d <days> -l <path>

  • ./purge_mongo.sh --mldata_db <ml database> -d <days> -l <path>

  • ./purge_mongo.sh --data_db <abs_database> --mldata_db <ml database> -d <days> -l <path>

For example, the following command dumps data older than 80 days from ml_database into a /tmp directory and deletes the data. /tmp is used as an example reference here, you can substitute /tmp with any other directory path in your environment.

./purge_mongo.sh --mldata_db abs_mldata -d 80 -l /tmp
Starting the purge mongo tool
Storing abs ml data from mongo at /tmp/ml_mongo_data.2020-11-16-06-12-14
This will delete the documents in  abs_mldata database that are older than 80 days.
Are you sure (yes/no): yes
Deleting the documents in  abs_mldata database that are older than 80 days.
Please see /opt/pingidentity/abs/logs/purge/purge.log.2020-11-16-06-12-42 for more details

By default, the script dumps all data and then removes processed data older than seven days.

In case there is a failure in execution of purge_mongo.sh the script exits. You can retry executing the script. The execution details are logged in/<pi-install path>/pingidentity/abs/logs/purge/directory.