PingIntelligence for APIs Dashboard uses index lifecycle management (ILM) policy support of Elasticsearch to roll over time-series data.
Rolling over the time-series data is important to maintain a low latency during search operations. The ILM policy allows for an automatic rollover of index based on time or size of data.
ILM policy for automatic rollover index works in Elasticsearch with X-Pack.
Configure automatic rollover index
You can configure the path to the ILM policy in
es.index.dashboard.activity.ilm.policy
property in
dashboard/config/dashboard.properties file. The ILM policy
file should be a valid JSON. Following is a sample ilm.json
file
available in the dashboard/config directory. Leave the value of
es.index.dashboard.activity.ilm.policy
property empty if you do
not wish to use ILM policy.
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "30GB",
"max_age": "30d"
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "30d",
"actions": {
"shrink": {
"number_of_shards": 1
},
"readonly": {},
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "90d",
"actions": {
"freeze": {},
"set_priority": {
"priority": 0
}
}
}
}
}
}
Policy phases
The ILM policy is divided into three phases:
hot
- In thehot
phase of the policy, the index is actively used to read and write data. The index remains in thehot
phase till the defined policy age or if the index reaches the maximum size. After the index reaches the age or size, it is rolled over and new index is created.Configure the
max_age
andmax_size
of the rollover index. The index is rolled over based on which value among the size and age is triggered first.warm
- In thewarm
phase of the policy, no new data is written to the index, however, it may be more frequently queried for searching data. The index next moves to thecold
phase.Configure the
min_age
of the index for thewarm
phase.cold
- In thecold
phase, index is neither written to or read from. In thecold
phase of policy, you can move the index to a low cost storage device.Configure the
min_age
of the index for thecold
phase.
Priority
After an Elasticsearch restart, indices are reloaded back into memory in sequence
according to priority. The index with the highest priority is loaded first. In the
above sample JSON, the hot
phase with priority 100 is of the
highest priority. The hot
index will be loaded into memory first.
The warm
phase with a priority number 50 is second in priority. The
warm
index will be loaded into memory after the
hot
index. Use a positive integer number to set the
priority.