Unassigned shards in Elasticsearch 7 and 8
There are multiple reasons why shards might get unassigned, ranging from misconfigured allocation settings to lack of disk space.
To reassign all unassigned shards in Elasticsearch, you can use the following steps:
STEPS:
- Check the status of unassigned shards
- Check cluster allocation explanation
- Automatic reallocation
- Force reallocation of shards
Check current cluster status
curl -XGET -u elastic:password http://172.16.0.1:9200/_cluster/health?pretty=true
{
"cluster_name" : "devcluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 24,
"active_shards" : 24,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 26,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 48.0
}
Check the status of unassigned shards:
First, you need to identify which shards are unassigned. You can do this by using the _cat/shards API. This will give you a list of all shards in your cluster along with their status.
curl -XGET -u elastic:password http://172.16.0.1:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason
.monitoring-kibana-7-2024.06.27 0 p STARTED
.monitoring-kibana-7-2024.06.27 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2024.05.29-000001 0 p STARTED
.ds-.logs-deprecation.elasticsearch-default-2024.05.29-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.apm-agent-configuration 0 p STARTED
.apm-agent-configuration 0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_7.17.13_001 0 p STARTED
.kibana_7.17.13_001 0 r UNASSIGNED CLUSTER_RECOVERED
.apm-custom-link 0 p STARTED
.apm-custom-link 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.25 0 p STARTED
.monitoring-es-7-2024.06.25 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.24 0 p STARTED
.monitoring-kibana-7-2024.06.24 0 r UNASSIGNED CLUSTER_RECOVERED
.ds-ilm-history-5-2024.05.29-000001 0 p STARTED
.ds-ilm-history-5-2024.05.29-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_task_manager_7.17.13_001 0 p STARTED
.kibana_task_manager_7.17.13_001 0 r UNASSIGNED CLUSTER_RECOVERED
.tasks 0 p STARTED
.tasks 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.22 0 p STARTED
.monitoring-kibana-7-2024.06.22 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.26 0 p STARTED
.monitoring-kibana-7-2024.06.26 0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_security_session_1 0 p UNASSIGNED INDEX_CREATED
.kibana_security_session_1 0 r UNASSIGNED REPLICA_ADDED
.monitoring-kibana-7-2024.06.25 0 p STARTED
.monitoring-kibana-7-2024.06.25 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.27 0 p STARTED
.monitoring-es-7-2024.06.27 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.21 0 p STARTED
.monitoring-kibana-7-2024.06.21 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.26 0 p STARTED
.monitoring-es-7-2024.06.26 0 r UNASSIGNED CLUSTER_RECOVERED
.geoip_databases 0 p STARTED
.geoip_databases 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.23 0 p STARTED
.monitoring-kibana-7-2024.06.23 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.23 0 p STARTED
.monitoring-es-7-2024.06.23 0 r UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.17.13-000001 0 p STARTED
.kibana-event-log-7.17.13-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.22 0 p STARTED
.monitoring-es-7-2024.06.22 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.21 0 p STARTED
.monitoring-es-7-2024.06.21 0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.24 0 p STARTED
.monitoring-es-7-2024.06.24 0 r UNASSIGNED CLUSTER_RECOVERED
.security-7 0 p STARTED
.security-7 0 r UNASSIGNED CLUSTER_RECOVERED
Check cluster allocation explanation:
To understand why shards are unassigned, you can use the _cluster/allocation/explain API. This will provide a detailed explanation of the allocation decisions.
curl -XGET -u elastic:password http://172.16.0.1:9200/_cluster/allocation/explain?pretty"
{
"node_id" : "vO3fSByBTxOCFzNsql-95g",
"node_name" : "dev01",
"transport_address" : "172.16.0.2:9300",
"node_attributes" : {
"ml.machine_memory" : "6067675136",
"ml.max_open_jobs" : "512",
"xpack.installed" : "true",
"ml.max_jvm_size" : "2147483648",
"transform.node" : "true"
},
"node_decision" : "no",
"weight_ranking" : 2,
"deciders" : [
{
"decider" : "enable",
"decision" : "NO",
"explanation" : "no allocations are allowed due to cluster setting [cluster.routing.allocation.enable=none]"
}
]
},
{
"node_id" : "kXPPZQDzTBu4ClLeWO95qQ",
"node_name" : "dev02",
"transport_address" : "172.16.0.3:9300",
"node_attributes" : {
"ml.machine_memory" : "6067666944",
"ml.max_open_jobs" : "512",
"xpack.installed" : "true",
"ml.max_jvm_size" : "2147483648",
"transform.node" : "true"
},
"node_decision" : "no",
"weight_ranking" : 3,
"deciders" : [
{
"decider" : "enable",
"decision" : "NO",
"explanation" : "no allocations are allowed due to cluster setting [cluster.routing.allocation.enable=none]"
}
Automatic reallocation:
Sometimes, cluster settings might prevent the allocation of shards. You can check your current cluster settings with:
curl -XGET -u elastic:password http://172.16.0.1:9200/_cluster/settings?pretty"
{
"peristent" : {
"cluster" : {
"routing" : {
"allocation" : {
"enable" : "none"
}
}
}
}
}
If shard allocation is disabled just like above, you can enable it by updating the cluster settings:
curl -XPUT -u elastic:password http://172.16.0.1:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
"persistent": {
"cluster.routing.allocation.enable": "all"
}
}'
Force reallocation of shards:
If you want to manually allocate an unassigned shard to a specific node, you can use the _cluster/reroute API. Here’s how to force the allocation of a stale primary shard:
curl -XPOST -u elastic:password http://172.16.0.1:9200/_cluster/reroute" -H 'Content-Type: application/json' -d '{
"commands": [
{
"allocate_stale_primary": {
"index": "index_name",
"shard": shard_number,
"node": "node_name",
"accept_data_loss": true
}
}
]
}'
Replace index_name, shard_number, and node_name with the appropriate values.
index: The name of the index to which the shard belongs.
shard: The shard number.
node: The name of the node to which you want to assign the shard.
accept_data_loss: Set to true if you want to allocate a stale primary shard, acknowledging that data loss might occur.
To allocate an unassigned replica shard, you can use:
curl -XPOST -u elastic:password http://172.16.0.1:9200/_cluster/reroute" -H 'Content-Type: application/json' -d '{
"commands": [
{
"allocate_replica": {
"index": "index_name",
"shard": shard_number,
"node": "node_name"
}
}
]
}'
To get the shard number and index name of unassigned shards in Elasticsearch, you can use the _cat/shards API. This API provides detailed information about all the shards in your cluster, including their index names, shard numbers, and current states.
Additional Steps and Considerations:
Check Node Availability: Ensure all nodes are up and running. Unassigned shards might be due to nodes being down or unreachable.
Disk Space: Make sure there is enough disk space on the nodes. Elasticsearch won’t allocate shards to nodes running low on disk space.
By following these steps, you should be able to diagnose and resolve issues related to unassigned shards in your Elasticsearch cluster.
Comments