Migrating Elasticsearch from RHEL 7.9 to 8

After migrating MongoDB, migrate Elasticsearch from RHEL 7.9 to RHEL 8.

Before you begin

Note the documents count before migration to compare the data after migration is complete:

curl -X GET "https://elasticsearch_ip:9200/_cat/indices?v" -u "elastic:changeme" -k

About this task

Elasticsearch installation is allowed for non-root user (ec2-user).

To migrate Elasticsearch:

Steps

Create a new RHEL 8 instance with 8 core CPU, 16 GB, 1 TB hard disk drive (HDD).

Download and install JAVA 11.0.2 and set JAVA_HOME with the following command:

export JAVA_HOME=/home/ec2-user/pingidentity/java
export PATH=$JAVA_HOME/bin:$PATH

Download the Elasticsearch build version that you’re migrating from.
Untar the Elasticsearch build by maintaining the same directory structure as the existing Elasticsearch:
```
/opt/pingidentity/elasticsearch
```
Copy elasticsearch.yml, elasticsearch_key.pem, elasticsearch_cert.pem, and elasticsearch.keystore from RHEL 7.9 Elasticsearch to the corresponding path in the RHEL 8 Elasticsearch instance.

If the path is different, you’ll need to update the path in the .yml file.

Start Elasticsearch.

./bin/elasticsearch -d -p ./logs/elasticsearch.pid

To verify that Elasticsearch started successfully, monitor the logs.
Stop both the RHEL 7 and RHEL 8 Elasticsearch process.
Copy the complete data/ folder from the RHEL 7 instance to the new RHEL 8 instance.
Start Elasticsearch in the RHEL 8 instance.
Verify the documents count.
Point the webgui and dataengine to the new Elasticsearch IP by changing the Elasticsearch IP in webgui.properties and dataengine.properties.

Start the PingIntelligence 5.1 components that point to RHEL 8 MongoDB and Elasticsearch:

Make sure that the API Behavioral Security (ABS) and API Publish configurations are pointing to the new mongo replica set (RHEL 8).
Make sure that webgui and dataengine are pointing to the new RHEL 8 Elasticsearch.
Start the PingIntelligence components in the following order:
1. API Security Enforcer (ASE)
2. ABS
3. API Publish
4. Data engine
5. Web GUI
To verify the data, access the Dashboard and verify the transactions count, indicators of attack (IOA), and discovered APIs.

Make an h2-backup.

Create an h2backup.sh file inside the webgui/bin folder with the following data:

#!/bin/bash
DATABASE_USER="sa"
DATABASE_PASSWORD="changeme changeme"
DATABASE_URL="jdbc:h2:ssl://localhost/webgui_data;CIPHER=AES"

H2_JAR_PATH="/home/ec2-user/pingidentity/webgui/lib/external/h2-*.jar"
TRUST_STORE_PATH="/home/ec2-user/pingidentity/webgui/config/webgui.jks"
OUTPUT_DIRECTORY="/home/ec2-user/pingidentity/webgui/data/h2-backup"

# Function to display an error message and exit with a non-zero status
function exit_with_error() {
    echo "Error: $1"
    exit 1
}

# Check if required environment variables are set
if [ -z "$DATABASE_USER" ] || [ -z "$DATABASE_PASSWORD" ] || [ -z "$DATABASE_URL" ] || [ -z "$H2_JAR_PATH" ] || [ -z "$TRUST_STORE_PATH" ] || [ -z "$OUTPUT_DIRECTORY" ]; then
    exit_with_error "One or more required environment variables are not set."
fi

# Ensure the H2 JAR file exists
if ! ls $H2_JAR_PATH > /dev/null 2>&1; then
    exit_with_error "H2 JAR file not found at '$H2_JAR_PATH'."
fi

# Ensure the trust store file exists
if [ ! -f "$TRUST_STORE_PATH" ]; then
    exit_with_error "Trust store file not found at '$TRUST_STORE_PATH'."
fi

# Ensure the output directory exists
if [ ! -d "$OUTPUT_DIRECTORY" ]; then
    exit_with_error "Output directory '$OUTPUT_DIRECTORY' not found."
fi

# List of tables to export
TABLES=("API_GROUP" "API_GROUP_ASSOCIATION" "API_STATE" "ASE_API" "CLIENT_VISIBILITY" "DISCOVERY_API" "DISCOVERY_METADATA" "USER" "USER_SESSION")

for TABLE_NAME in ${TABLES[@]}; do
    # Generate the export query for the current table
    EXPORT_QUERY="SELECT * FROM $TABLE_NAME;"

    csv_path="$OUTPUT_DIRECTORY/$TABLE_NAME.csv"

    # Export the data of the current table as CSV
    java -Djavax.net.ssl.trustStore=$TRUST_STORE_PATH -cp $H2_JAR_PATH org.h2.tools.Shell -url "$DATABASE_URL" -user "$DATABASE_USER" -password "$DATABASE_PASSWORD" -sql "CALL CSVWRITE('$csv_path', '$EXPORT_QUERY')"

    # Check the exit status of the java command
    if [ $? -ne 0 ]; then
        exit_with_error "Failed to export data for table '$TABLE_NAME'."
    fi
done

echo "Data export completed successfully."

Update the paths for the following variables:

H2_JAR_PATH
TRUST_STORE_PATH
OUTPUT_DIRECTORY

Create the h2-backup directory under data/.
```
mkdir webgui/data/h2-backup
```
Give executable permissions to the h2backup.sh file.
Run the ./bin/h2backup.sh script to create the h2-backup folder inside the webgui/data directory.

After successful completion of the script run, verify the backup data by checking the .csv files created in the h2-backup folder.

ls webgui/data/h2-backup/
API_GROUP.csv  API_GROUP_ASSOCIATION.csv  API_STATE.csv  ASE_API.csv  CLIENT_VISIBILITY.csv  DISCOVERY_API.csv  DISCOVERY_METADATA.csv  USER.csv  USER_SESSION.csv

Change "discovery_source","abs" to "discovery_source","ABS" in the DISCOVERY_METADATA.csv file, and save the file.
```
"seedDataInitAdmin","2023-09-07 17:00:31.365","discovery_source","ABS"
```

Copy and keep the h2-backup folder separate.

The backup folder will be used in Upgrading PingIntelligence.

Troubleshooting:

If the error below is observed in the webgui admin.log, restart the Dashboard to resolve the issue.

error c.p.p.c.ElasticSearchClientConfiguration [https-jsse-nio-0.0.0.0-8030-exec-3] FailureListener | Error with Node: [host=https://10.96.6.196:9200]
2023-08-22 06:22:00 error c.p.p.w.s.d.MainDashboardActionsServiceImpl [https-jsse-nio-0.0.0.0-8030-exec-3] Failed to Fetch Attack Lists Count from Elasticsearch. Error: pi4api.commons | Error while executing Elasticsearch msearch Query Type: bool.
com.pingidentity.pingintelligence.exception.PIOperationsException: pi4api.commons | Error while executing Elasticsearch msearch Query Type: bool

Next steps

Complete the steps in Upgrading PingIntelligence.