Elasticsearch Operator doesn't allow to reduce size of existing persistent volume claims. The instruction is demonstrating workaround by creating a new elasticsearch `nodeSet` followed by data migration. Elasticsearch operator takes care of resource usage, it conducts the procedure by cycle:
- create a new elasticsearch node
- drain one of existing elasticsearch nodes
- terminate drained elasticsearch node
and repeats until all nodes of previous nodeSet terminated. The procedure require resources enough to accommodation 2 additional elasticsearch pods. Instruction was tested on OpenShift cluster and Elasticsearch v8.18 managed by eck-operator-3.0.0.
Step 1. Check current status of elasticsearch cluster
Ensure elasticsearch cluster is in "Ready" state:
oc get es -n instana-elastic
example of expected output:
NAME HEALTH NODES VERSION PHASE AGE
instana green 3 8.18.7 Ready 16d
Ensure elasticsearch cluster status is green:
ELASTICPASS=`oc -n instana-elastic get secret instana-es-elastic-user --template='{{ index .data "elastic" | base64decode }}'`
alias es="oc -n instana-elastic exec service/instana-es-http -- curl -uelastic:$ELASTICPASS"
es -s -X GET "localhost:9200/_cluster/health?filter_path=status,*_shards&pretty"
example of expected output:
Defaulted container "elasticsearch" out of: elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
{
"status" : "green",
"active_primary_shards" : 32,
"active_shards" : 58,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"unassigned_primary_shards" : 0,
"delayed_unassigned_shards" : 0
}
Step 2. Edit elasticsearch CRD
Once the Elasticsearch is "ready" and "green" we can configure Elasticsearch CRD:
oc edit es -n instana-elastic
In the custom resource definition set required size and change name of the nodeSet (in the example "default"). We can't reduce PVC, but can create dataSet for new nodes and operator will move the data sequentially:
. . .
nodeSets:
- config:
. . .
count: 3
name: default <------ Change name to for example "newdefault"
. . .
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
resources:
requests:
storage: 50Gi <------ Set required PVC size
. . .
Save and exit. The procedure will trigger cluster re-shape: all pods will be re-created one by one data migrated. On clusters with large size the process will take long time. No other steps required, data migration if managed by elasticsearch operator.
---
Relevant documentation:
- https://www.ibm.com/docs/en/instana-observability/1.0.315?topic=installing-elasticsearch
#Administration
#Self-Hosted