Purge data from Elasticsearch

To manage storage on the Dashboard server, you can either archive or purge Elasticsearch data. PingIntelligence provides a purge script to remove older Elasticsearch data. WARNING: When the purge script is run, all files are permanently deleted from the Elasticsearch data directory. Hence it is recommended to take a backup of Elasticsearch documents before proceeding with the purge.

Run the purge script, on the dashboard engine command line. The number of days specified should be between 1-365 days.

/opt/pingidentity/dashboard/util/purge_elasticsearch.sh -d 3

In the following example, purge_elasticsearch.sh deletes all files older than 3 days. Here is a sample output:

/opt/pingidentity/dashboard/util/purge_elasticsearch.sh -d 3
This will delete the data in elastic search which is older than 3 days.
Are You sure(yes/no):yes
2017-04-17 11:13:07 INFO Starting purge with options, days : 3 path : /opt/poc/pingidentity/dashboard/config/dashboard.properties

To delete all data and Elasticsearch templates, use the following:

curl -s https://<elasticsearch_ip_address>:<port>/_all -X DELETE -u elastic

When you use the -X DELETE option, the system goes back to a fresh installation state. NOTE: Purge for Elasticsearch runs in the background. Documents are not deleted immediately after purge_elasticsearch.sh execution. Elasticsearch deletes purged documents with a lag of 5 minutes.

The following example illustrates deletion of Elasticsearch records older than 15 days. The Number of Records Purged : null is an expected message due to the time lag in actual deletion.

[xxxxxxxxx@T5-03 dashboard]$ ./util/purge_elasticsearch.sh -d 15
This will delete the data in elasticsearch cluster which are older than 15 days.
Are You sure(yes/no):yes
Starting Elasticsearch purge
2020-04-09 03:16:44 INFO   Starting purge with options, days : 15 path : /home/xxxxxxxx/pingidentity/dashboard/config/dashboard.properties
2020-04-09 03:16:45 INFO   API's Loaded from elasticsearch : [app54, app58, app63, app8, app2, app3, app66, app74, app79, app77]
2020-04-09 03:16:45 INFO   Purging data for global indice activity-api
2020-04-09 03:16:45 INFO   Number of Records Purged :  null
2020-04-09 03:16:45 INFO   Purging data for global indice activity-api-key
2020-04-09 03:16:45 INFO   Number of Records Purged :  null
2020-04-09 03:16:45 INFO   Purging data for global indice activity-token
2020-04-09 03:16:45 INFO   Number of Records Purged :  null
.
.

It is recommended to run purge_elasticsearch.sh during lean API traffic periods.