PingIntelligence

Purging data from Elasticsearch

To manage storage on the Dashboard server, you can either archive or purge Elasticsearch data.

About this task

PingIntelligence provides a purge script to remove older Elasticsearch data.

When the purge script is run, all files are permanently deleted from the Elasticsearch data directory. Hence it is recommended to make a backup of Elasticsearch documents before proceeding with the purge.

Steps

  1. Run the purge script on the Dashboard engine command line.

    The number of days specified should be between 1-365 days.

    /opt/pingidentity/dashboard/util/purge_elasticsearch.sh -d 3

    Example:

    In the following example, purge_elasticsearch.sh deletes all files older than 3 days. Below is a sample output:

    /opt/pingidentity/dashboard/util/purge_elasticsearch.sh -d 3
This will delete the data in elastic search which is older than 3 days.
Are You sure(yes/no):yes
2017-04-17 11:13:07 INFO Starting purge with options, days : 3 path : /opt/poc/pingidentity/dashboard/config/dashboard.properties

  2. To delete all data and Elasticsearch templates, use the following:

    curl -s https://<elasticsearch_ip_address>:<port>/_all -X DELETE -u elastic

    Example:

    The following example illustrates deletion of Elasticsearch records older than 15 days. The Number of Records Purged : null is an expected message due to the time lag in actual deletion.

    [xxxxxxxxx@T5-03 dashboard]$ ./util/purge_elasticsearch.sh -d 15
This will delete the data in elasticsearch cluster which are older than 15 days.
Are You sure(yes/no):yes
Starting Elasticsearch purge
2020-04-09 03:16:44 INFO   Starting purge with options, days : 15 path : /home/xxxxxxxx/pingidentity/dashboard/config/dashboard.properties
2020-04-09 03:16:45 INFO   API's Loaded from elasticsearch : [app54, app58, app63, app8, app2, app3, app66, app74, app79, app77]
2020-04-09 03:16:45 INFO   Purging data for global indice activity-api
2020-04-09 03:16:45 INFO   Number of Records Purged :  null
2020-04-09 03:16:45 INFO   Purging data for global indice activity-api-key
2020-04-09 03:16:45 INFO   Number of Records Purged :  null
2020-04-09 03:16:45 INFO   Purging data for global indice activity-token
2020-04-09 03:16:45 INFO   Number of Records Purged :  null
.
.

    Result:

    When you use the -X DELETE option, the system goes back to a fresh installation state.

    Purge for Elasticsearch runs in the background. Documents are not deleted immediately after purge_elasticsearch.sh execution. Elasticsearch deletes purged documents with a lag of 5 minutes. It is recommended to run purge_elasticsearch.sh during lean API traffic periods.