Elasticsearch Index Management
I have been using elasticsearch from its early version. Index management is the task that was not automated in the older version & a simple script for deleting indexes are taken place. Now Elasticsearch versions are more mature & come with nice Index Lifecycle Management.
ILM & ISM
Opensource elasticsearch follows “Index Lifecycle Management”, and its plugin follows the _ilm prefix.
If you try the same with Amazon managed Elasticsearch (called OpenSearch nowadays), it uses ism (Index State Management).
Both are very similar, a small command change is observed below for same archiving task:
Simple Archive Setup
Elastic search Policy works on states as shown earlier, you can be transited from one state to another with or without action mentioned on it.
Below shows the command for archiving in AWS managed Elasticsearch / opensearch where version deployed is opendistro.
PUT _opendistro/_ism/policies/test-archive
{
"policy": {
"description": "A simple default policy that archives logs after 15 days.",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "15d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
],
"ism_template": {
"index_patterns": ["logs-to-be-archive-*"],
"priority": 100
}}
}
Below shows command for the opensource elastic search
PUT _ilm/policy/test-archive
{
"policy": {
"_meta": {
"description": "A simple default policy that archives logs after 15 days.",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"delete": {
"min_age": "15d",
"actions": {
"delete": {}
}
}
}
}
}PUT _index_template/timeseries_template
{
"index_patterns": ["logs-to-be-archive-*"],
"template": {
"settings": {
"index.lifecycle.name": "test-archive"
}
}
}
There are more complex index management is possible. What is all supported is detailed here.
Error:{“type”:”index_create_block_exception”,”reason”:”blocked by: [FORBIDDEN/10/cluster create-index blocked (api)];”}
If you get this error, it means that there is no disk space on the cluster. Remove indexes & run the command again.
Reference:
