--- title: Migrating Elasticsearch from RHEL 7.9 to 8 description: After migrating MongoDB, migrate Elasticsearch from RHEL 7.9 to RHEL 8. component: pingintelligence version: 5.2 page_id: pingintelligence:installing_pingintelligence_for_apis:pingintelligence_migrating_elasticsearch canonical_url: https://docs.pingidentity.com/pingintelligence/5.2/installing_pingintelligence_for_apis/pingintelligence_migrating_elasticsearch.html revdate: May 6, 2024 section_ids: before-you-begin: Before you begin about-this-task: About this task steps: Steps troubleshooting: Troubleshooting: next-steps: Next steps --- # Migrating Elasticsearch from RHEL 7.9 to 8 After migrating MongoDB, migrate Elasticsearch from RHEL 7.9 to RHEL 8. ## Before you begin Note the documents count before migration to compare the data after migration is complete: ```shell curl -X GET "https://elasticsearch_ip:9200/_cat/indices?v" -u "elastic:changeme" -k ``` ## About this task | | | | - | --------------------------------------------------------------------- | | | Elasticsearch installation is allowed for non-root user (`ec2-user`). | To migrate Elasticsearch: ## Steps 1. Create a new RHEL 8 instance with 8 core CPU, 16 GB, 1 TB hard disk drive (HDD). 2. [Download and install JAVA 11.0.2](https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz) and set `JAVA_HOME` with the following command: ``` export JAVA_HOME=/home/ec2-user/pingidentity/java export PATH=$JAVA_HOME/bin:$PATH ``` 3. [Download the Elasticsearch build version that you're migrating from](https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.13.4-linux-x86_64.tar.gz). 4. Untar the Elasticsearch build by maintaining the same directory structure as the existing Elasticsearch: ``` /opt/pingidentity/elasticsearch ``` 5. Copy `elasticsearch.yml`, `elasticsearch_key.pem`, `elasticsearch_cert.pem`, and `elasticsearch.keystore` from RHEL 7.9 Elasticsearch to the corresponding path in the RHEL 8 Elasticsearch instance. | | | | - | ---------------------------------------------------------------------------- | | | If the path is different, you'll need to update the path in the `.yml` file. | 6. Start Elasticsearch. ``` ./bin/elasticsearch -d -p ./logs/elasticsearch.pid ``` 7. To verify that Elasticsearch started successfully, monitor the logs. 8. Stop both the RHEL 7 and RHEL 8 Elasticsearch process. 9. Copy the complete `data/` folder from the RHEL 7 instance to the new RHEL 8 instance. 10. Start Elasticsearch in the RHEL 8 instance. 11. Verify the documents count. 12. Point the `webgui` and `dataengine` to the new Elasticsearch IP by changing the Elasticsearch IP in `webgui.properties` and `dataengine.properties`. 13. Start the PingIntelligence 5.1 components that point to RHEL 8 MongoDB and Elasticsearch: 1. Make sure that the API Behavioral Security (ABS) and API Publish configurations are pointing to the new `mongo` replica set (RHEL 8). 2. Make sure that `webgui` and `dataengine` are pointing to the new RHEL 8 Elasticsearch. 3. Start the PingIntelligence components in the following order: 1. API Security Enforcer (ASE) 2. ABS 3. API Publish 4. Data engine 5. Web GUI 4. To verify the data, access the Dashboard and verify the transactions count, indicators of attack (IOA), and discovered APIs. 5. Make an `h2-backup`. 1. Create an `h2backup.sh` file inside the `webgui/bin` folder with the following data: ``` #!/bin/bash DATABASE_USER="sa" DATABASE_PASSWORD="changeme changeme" DATABASE_URL="jdbc:h2:ssl://localhost/webgui_data;CIPHER=AES" H2_JAR_PATH="/home/ec2-user/pingidentity/webgui/lib/external/h2-*.jar" TRUST_STORE_PATH="/home/ec2-user/pingidentity/webgui/config/webgui.jks" OUTPUT_DIRECTORY="/home/ec2-user/pingidentity/webgui/data/h2-backup" # Function to display an error message and exit with a non-zero status function exit_with_error() { echo "Error: $1" exit 1 } # Check if required environment variables are set if [ -z "$DATABASE_USER" ] || [ -z "$DATABASE_PASSWORD" ] || [ -z "$DATABASE_URL" ] || [ -z "$H2_JAR_PATH" ] || [ -z "$TRUST_STORE_PATH" ] || [ -z "$OUTPUT_DIRECTORY" ]; then exit_with_error "One or more required environment variables are not set." fi # Ensure the H2 JAR file exists if ! ls $H2_JAR_PATH > /dev/null 2>&1; then exit_with_error "H2 JAR file not found at '$H2_JAR_PATH'." fi # Ensure the trust store file exists if [ ! -f "$TRUST_STORE_PATH" ]; then exit_with_error "Trust store file not found at '$TRUST_STORE_PATH'." fi # Ensure the output directory exists if [ ! -d "$OUTPUT_DIRECTORY" ]; then exit_with_error "Output directory '$OUTPUT_DIRECTORY' not found." fi # List of tables to export TABLES=("API_GROUP" "API_GROUP_ASSOCIATION" "API_STATE" "ASE_API" "CLIENT_VISIBILITY" "DISCOVERY_API" "DISCOVERY_METADATA" "USER" "USER_SESSION") for TABLE_NAME in ${TABLES[@]}; do # Generate the export query for the current table EXPORT_QUERY="SELECT * FROM $TABLE_NAME;" csv_path="$OUTPUT_DIRECTORY/$TABLE_NAME.csv" # Export the data of the current table as CSV java -Djavax.net.ssl.trustStore=$TRUST_STORE_PATH -cp $H2_JAR_PATH org.h2.tools.Shell -url "$DATABASE_URL" -user "$DATABASE_USER" -password "$DATABASE_PASSWORD" -sql "CALL CSVWRITE('$csv_path', '$EXPORT_QUERY')" # Check the exit status of the java command if [ $? -ne 0 ]; then exit_with_error "Failed to export data for table '$TABLE_NAME'." fi done echo "Data export completed successfully." ``` 2. Update the paths for the following variables: ``` H2_JAR_PATH TRUST_STORE_PATH OUTPUT_DIRECTORY ``` 3. Create the `h2-backup` directory under `data/`. ``` mkdir webgui/data/h2-backup ``` 4. Give executable permissions to the `h2backup.sh` file. 5. Run the `./bin/h2backup.sh` script to create the `h2-backup` folder inside the `webgui/data` directory. 6. After successful completion of the script run, verify the backup data by checking the `.csv` files created in the `h2-backup` folder. ``` ls webgui/data/h2-backup/ API_GROUP.csv API_GROUP_ASSOCIATION.csv API_STATE.csv ASE_API.csv CLIENT_VISIBILITY.csv DISCOVERY_API.csv DISCOVERY_METADATA.csv USER.csv USER_SESSION.csv ``` 7. Change `"discovery_source","abs"` to `"discovery_source","ABS"` in the `DISCOVERY_METADATA.csv` file, and save the file. ``` "seedDataInitAdmin","2023-09-07 17:00:31.365","discovery_source","ABS" ``` 8. Copy and keep the `h2-backup` folder separate. The backup folder will be used in [Upgrading PingIntelligence](pingintelligence_upgrading.html). ### Troubleshooting: If the error below is observed in the `webgui admin.log`, restart the Dashboard to resolve the issue. ``` error c.p.p.c.ElasticSearchClientConfiguration [https-jsse-nio-0.0.0.0-8030-exec-3] FailureListener | Error with Node: [host=https://10.96.6.196:9200] 2023-08-22 06:22:00 error c.p.p.w.s.d.MainDashboardActionsServiceImpl [https-jsse-nio-0.0.0.0-8030-exec-3] Failed to Fetch Attack Lists Count from Elasticsearch. Error: pi4api.commons | Error while executing Elasticsearch msearch Query Type: bool. com.pingidentity.pingintelligence.exception.PIOperationsException: pi4api.commons | Error while executing Elasticsearch msearch Query Type: bool ``` ## Next steps Complete the steps in [Upgrading PingIntelligence](pingintelligence_upgrading.html).