This repository has been archived by the owner on Feb 22, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 16.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Harden aspects of the elasticsearch chart
* Added configmap to explicitly provide cluster configurations and scripts * Replace depreciating `ES_HEAP_SIZE` with `ES_JAVA_OPTS` to position for ES v5 support * Removed alpha storage class operators * Removed catastrophic liveness probe checking entire clusters health * Readiness probe now inspects local node health * Added termination grace period (defaults to 15m) to allow pre-stop-script.sh time to gracefully migrate shards * Added init container to configure `vm.max_map_count` * Updated elasticsearch.yaml: * Added `PROCESSOR` configuration to prevent large cluster garbage collection issues leading to node eviction * Added configurable gateway defaults to help avoid a split brain, requiring two masters online and in consensus before recovery can continue * Updated pre-stop-script.sh: * Check `v1beta1` `statefulset` endpoint * Evalute `.spec.replicas` for statefulset desired size * Clear `_cluster/settings` ip exclusion prior to shutdown to avoid a possible (random) ip match scenario on expansion of the clsuter * Data nodes now use default storage class if once is not specified
- Loading branch information
Showing
8 changed files
with
258 additions
and
93 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,15 @@ | ||
| name: elasticsearch | ||
| home: https://www.elastic.co/products/elasticsearch | ||
| version: 0.1.5 | ||
| version: 0.1.6 | ||
| description: Flexible and powerful open source, distributed real-time search and analytics engine. | ||
| icon: https://static-www.elastic.co/assets/blteb1c97719574938d/logo-elastic-elasticsearch-lt.svg | ||
| sources: | ||
| - https://www.elastic.co/products/elasticsearch | ||
| - https://github.com/jetstack/elasticsearch-pet | ||
| - https://github.com/giantswarm/kubernetes-elastic-stack | ||
| - https://github.com/GoogleCloudPlatform/elasticsearch-docker | ||
| maintainers: | ||
| - name: Christian Simon | ||
| email: [email protected] | ||
| - name: Michael Haselton | ||
| email: [email protected] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
111 changes: 111 additions & 0 deletions
111
incubator/elasticsearch/templates/elasticsearch-configmap.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}" | ||
| labels: | ||
| app: "{{ printf "%s-%s" .Release.Name .Values.Name | trunc 24 }}" | ||
| chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" | ||
| release: "{{ .Release.Name }}" | ||
| heritage: "{{ .Release.Service }}" | ||
| data: | ||
| pre-stop-hook.sh: |- | ||
| #!/bin/bash | ||
| set -e | ||
| SERVICE_ACCOUNT_PATH=/var/run/secrets/kubernetes.io/serviceaccount | ||
| KUBE_TOKEN=$(<${SERVICE_ACCOUNT_PATH}/token) | ||
| KUBE_NAMESPACE=$(<${SERVICE_ACCOUNT_PATH}/namespace) | ||
| STATEFULSET_NAME=$(echo "${HOSTNAME}" | sed 's/-[0-9]*$//g') | ||
| INSTANCE_ID=$(echo "${HOSTNAME}" | grep -o '[0-9]*$') | ||
| echo "Prepare stopping of Pet ${KUBE_NAMESPACE}/${HOSTNAME} of StatefulSet ${KUBE_NAMESPACE}/${STATEFULSET_NAME} instance_id ${INSTANCE_ID}" | ||
| INSTANCES_DESIRED=$(curl -s \ | ||
| --cacert ${SERVICE_ACCOUNT_PATH}/ca.crt \ | ||
| -H "Authorization: Bearer $KUBE_TOKEN" \ | ||
| "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_PORT_443_TCP_PORT}/apis/apps/v1beta1/namespaces/${KUBE_NAMESPACE}/statefulsets/${STATEFULSET_NAME}/status" | jq -r '.spec.replicas') | ||
| echo "Desired instance count is ${INSTANCES_DESIRED}" | ||
| if [ "${INSTANCE_ID}" -lt "${INSTANCES_DESIRED}" ]; then | ||
| echo "No data migration needed" | ||
| exit 0 | ||
| fi | ||
| echo "Prepare to migrate data of the node" | ||
| NODE_STATS=$(curl -s -XGET 'http://localhost:9200/_nodes/stats') | ||
| NODE_IP=$(echo "${NODE_STATS}" | jq -r ".nodes[] | select(.name==\"${HOSTNAME}\") | .host") | ||
| echo "Move all data from node ${NODE_IP}" | ||
| curl -s -XPUT localhost:9200/_cluster/settings -d "{ | ||
| \"transient\" :{ | ||
| \"cluster.routing.allocation.exclude._ip\" : \"${NODE_IP}\" | ||
| } | ||
| }" | ||
| echo | ||
| echo "Wait for node to become empty" | ||
| DOC_COUNT=$(echo "${NODE_STATS}" | jq ".nodes[] | select(.name==\"${HOSTNAME}\") | .indices.docs.count") | ||
| while [ "${DOC_COUNT}" -gt 0 ]; do | ||
| NODE_STATS=$(curl -s -XGET 'http://localhost:9200/_nodes/stats') | ||
| DOC_COUNT=$(echo "${NODE_STATS}" | jq -r ".nodes[] | select(.name==\"${HOSTNAME}\") | .indices.docs.count") | ||
| echo "Node contains ${DOC_COUNT} documents" | ||
| sleep 1 | ||
| done | ||
| curl -s -XPUT localhost:9200/_cluster/settings -d "{ | ||
| \"transient\" :{ | ||
| \"cluster.routing.allocation.exclude._ip\" : \"\" | ||
| } | ||
| }" | ||
| echo | ||
| echo "Node clear to shutdown" | ||
| elasticsearch.yml: |- | ||
| node.data: ${NODE_DATA:true} | ||
| node.master: ${NODE_MASTER:true} | ||
| node.name: ${HOSTNAME} | ||
| # see https://github.com/kubernetes/kubernetes/issues/3595 | ||
| bootstrap.mlockall: ${BOOTSTRAP_MLOCKALL:false} | ||
| network.host: 0.0.0.0 | ||
| cloud: | ||
| kubernetes: | ||
| service: ${SERVICE} | ||
| namespace: ${KUBERNETES_NAMESPACE} | ||
| discovery: | ||
| type: kubernetes | ||
| zen: | ||
| minimum_master_nodes: 2 | ||
| # see https://github.com/elastic/elasticsearch-definitive-guide/pull/679 | ||
| processors: ${PROCESSORS:} | ||
| # avoid split-brain w/ a minimum consensus of two masters plus a data node | ||
| gateway.expected_master_nodes: ${EXPECTED_MASTER_NODES:2} | ||
| gateway.expected_data_nodes: ${EXPECTED_DATA_NODES:1} | ||
| gateway.recover_after_time: ${RECOVER_AFTER_TIME:5m} | ||
| gateway.recover_after_master_nodes: ${RECOVER_AFTER_MASTER_NODES:2} | ||
| gateway.recover_after_data_nodes: ${RECOVER_AFTER_DATA_NODES:1} | ||
| logging.yml: |- | ||
| # you can override this using by setting a system property, for example -Des.logger.level=DEBUG | ||
| es.logger.level: INFO | ||
| rootLogger: ${es.logger.level}, console | ||
| logger: | ||
| # log action execution errors for easier debugging | ||
| action: DEBUG | ||
| # reduce the logging for aws, too much is logged under the default INFO | ||
| com.amazonaws: WARN | ||
| appender: | ||
| console: | ||
| type: console | ||
| layout: | ||
| type: consolePattern | ||
| conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.