PingIntelligence

Automatic rollover index

PingIntelligence for APIs Dashboard uses Index Lifecycle Management (ILM) policy support of Elasticsearch to rollover time-series data. Rolling over the time-series data is important to maintain a low latency during search operations. The ILM policy allows for an automatic rollover of index based on time or size of data.

ILM policy for automatic rollover index works in Elasticsearch with X-Pack.

Configuring automatic rollover index

Configure the path to the ILM policy in es.index.dashboard.activity.ilm.policy property in dashboard/config/dashboard.properties file. The ILM policy file should be a valid JSON. Following is a sample ilm.json file available in the dashboard/config directory. Leave the value of es.index.dashboard.activity.ilm.policy property empty if you do not wish to use ILM policy.

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "30GB",
            "max_age": "30d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "readonly": {},
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "freeze": {},
          "set_priority": {
            "priority": 0
          }
        }
      }
    }
  }
}

Policy phases - The ILM policy is divided into three phases:

  • hot - In the hot phase of the policy, the index is actively used to read and write data. The index remains in the hot phase till the defined policy age or if the index reaches the maximum size. After the index reaches the age or size, it is rolled over and new index is created.

    Configure the max_age and max_size of the rollover index. The index is rolled over based on which value among the size and age is triggered first.

  • warm - In the warm phase of the policy, no new data is written to the index, however, it may be more frequently queried for searching data. The index next moves to the cold phase.

    Configure the min_age of the index for the warm phase.

  • cold - In the cold phase, index is neither written to or read from. In the cold phase of policy, you can move the index to a low cost storage device.

    Configure the min_age of the index for the cold phase.

Priority - After an Elasticsearch restart, indices are reloaded back into memory in sequence according to priority. Index with highest priority is loaded first. In the above sample JSON, the hot phase with priority 100 is of the highest priority. Hot index will be loaded into memory first. The warm phase with a priority number 50 is second in priority. Warm index will be loaded into memory after hot index. Use a positive integer number to set the priority.