Skip to content

helm: minikube example #4383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sougou merged 5 commits into from
Nov 25, 2018
Merged

helm: minikube example #4383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
59f3051
helm: minikube example
sougou Nov 23, 2018
28b14d9
helm: add support for schema and vschema
sougou Nov 24, 2018
440e157
helm: address review comments
sougou Nov 24, 2018
8632ec1
helm: k8s_secret -> secret
sougou Nov 24, 2018
d93381b
helm: move ApplySchema to keyspace
sougou Nov 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions helm/vitess/examples/minikube.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
topology:
cells:
- name: "zone1"
etcd:
replicas: 1
vtctld:
replicas: 1
vtgate:
replicas: 1
mysqlProtocol:
enabled: true
authType: "none"
keyspaces:
- name: "messagedb"
schema: |-
CREATE TABLE messages (
page BIGINT(20) UNSIGNED,
time_created_ns BIGINT(20) UNSIGNED,
message VARCHAR(10000),
PRIMARY KEY (page, time_created_ns)
) ENGINE=InnoDB
vschema: |-
{
"sharded": true,
"vindexes": {
"hash": {
"type": "hash"
}
},
"tables": {
"messages": {
"column_vindexes": [
{
"column": "page",
"name": "hash"
}
]
}
}
}
shards:
- name: "0"
tablets:
- type: "replica"
vttablet:
replicas: 2

etcd:
replicas: 1
resources:

vtctld:
serviceType: "NodePort"
resources:
Copy link
Copy Markdown
Member

@derekperkins derekperkins Nov 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work with empty resources? It looks really strange. I would either expect to see resources: {} or just set some lower resources than in the default chart.

Copy link
Copy Markdown
Contributor Author

@sougou sougou Nov 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is how they recommend you delete a key: https://github.com/helm/helm/blob/master/docs/chart_template_guide/values_files.md#deleting-a-default-key.

Copy link
Copy Markdown
Member

@derekperkins derekperkins Nov 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It still feels strange. You don't want to just set low resource usage? You should be able to spin up your cluster and see how much each pod/container is consuming.

Copy link
Copy Markdown
Contributor Author

@sougou sougou Nov 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting for @dkhenry to weigh in. He's the one who recommended null resources.

In terms of resource, the concern I have is that these pods burn a lot of CPU when they come up, but then become mostly idle. Later, their load would go up depending on what the users would try. In other words, I don't know how to size these pods.

Copy link
Copy Markdown
Member

@derekperkins derekperkins Nov 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CPU is less important since it just causes things to run slowly. It more the RAM constraints that I think are valuable, especially for someone wanting to know how big their VM needs to be for this example to come up.

I'm not against leaving null resources, but if we leave them null, it'd be helpful to have comments as to why they are null.

Copy link
Copy Markdown
Contributor

@dkhenry dkhenry Nov 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sougou For the example you can set Requests so people have an idea on how to size their VM

      requests:
        memory: "256Mi"
        cpu: "250m"
``

That would be for the tablets and gateways, and for the control containers you can cut down the ram a bit


vtgate:
serviceType: "NodePort"
resources:

vttablet:
resources:
mysqlResources:

pmm:
enabled: false
client:
resources:
server:
resources:

orchestrator:
enabled: false
resources:
6 changes: 3 additions & 3 deletions helm/vitess/templates/NOTES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{{- $cell := (index .Values.topology.cells 0).name -}}
{{- $proxyURL := printf "http://localhost:8001/api/v1/proxy/namespaces/%s" .Release.Namespace -}}
{{- $proxyURL := printf "http://localhost:8001/api/v1/namespaces/%s" .Release.Namespace -}}

Release name: {{.Release.Name}}

Expand All @@ -8,7 +8,7 @@ To access administrative web pages, start a proxy with:

Then use the following URLs:

vtctld: {{$proxyURL}}/services/vtctld:web/app/
vtgate: {{$proxyURL}}/services/vtgate-{{$cell}}:web/
vtctld: {{$proxyURL}}/services/vtctld:web/proxy/app/
vtgate: {{$proxyURL}}/services/vtgate-{{$cell}}:web/proxy/
{{ if $.Values.orchestrator.enabled }}orchestrator: {{$proxyURL}}/services/orchestrator:web/{{ end }}
{{ if $.Values.pmm.enabled }} pmm: {{$proxyURL}}/services/pmm:web/{{ end }}
135 changes: 135 additions & 0 deletions helm/vitess/templates/_keyspace.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
###################################
# keyspace initializations
###################################

{{- define "keyspace" -}}
{{- $cell := index . 0 -}}
{{- $keyspace := index . 1 -}}
{{- $defaultVtctlclient := index . 2 -}}
{{- $namespace := index . 3 -}}

# sanitize inputs for labels
{{- $keyspaceClean := include "clean-label" $keyspace.name -}}

{{- with $cell.vtctld -}}

# define image to use
{{- $vitessTag := .vitessTag | default $defaultVtctlclient.vitessTag -}}

{{- if $keyspace.schema }}
---
###################################
# ApplySchema Job
###################################
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $keyspaceClean }}-apply-schema
spec:
backoffLimit: 1
template:
spec:
restartPolicy: OnFailure
containers:
- name: apply-schema
image: "vitess/vtctlclient:{{$vitessTag}}"
volumeMounts:
{{ include "user-secret-volumeMounts" $defaultVtctlclient.secrets | indent 10 }}

command: ["bash"]
args:
- "-c"
- |
set -ex

VTCTLD_SVC=vtctld.{{ $namespace }}:15999
SECONDS=0
TIMEOUT_SECONDS=600
VTCTL_EXTRA_FLAGS=({{ include "format-flags-inline" $defaultVtctlclient.extraFlags }})

# poll every 5 seconds to see if vtctld is ready
until vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC ListAllTablets {{ $cell.name }} > /dev/null 2>&1; do
if (( $SECONDS > $TIMEOUT_SECONDS )); then
echo "timed out waiting for vtctlclient to be ready"
exit 1
fi
sleep 5
done

while true; do
if (( $SECONDS > $TIMEOUT_SECONDS )); then
echo "timed out waiting for master"
exit 1
fi

# wait for all shards to have a master
{{- range $shard := $keyspace.shards }}
master_alias=$(vtctlclient ${VTLCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC GetShard {{ $keyspace.name }}/{{ $shard.name }} | jq '.master_alias.uid')
if [ "$master_alias" == "null" -o "$master_alias" == "" ]; then
echo "no master for '{{ $keyspace.name }}/{{ $shard.name }}' yet, continuing to wait"
sleep 5
continue
fi
{{- end }}

break
done

vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC ApplySchema -sql "$(cat <<END_OF_COMMAND
{{ $keyspace.schema | indent 14}}
END_OF_COMMAND
)" {{ $keyspace.name }}
volumes:
{{ include "user-secret-volumes" (.secrets | default $defaultVtctlclient.secrets) | indent 8 }}
{{ end }}

{{- if $keyspace.vschema }}
---
###################################
# ApplyVSchema job
###################################
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $keyspaceClean }}-apply-vschema
spec:
backoffLimit: 1
template:
spec:
restartPolicy: OnFailure
containers:
- name: apply-vschema
image: "vitess/vtctlclient:{{$vitessTag}}"
volumeMounts:
{{ include "user-secret-volumeMounts" $defaultVtctlclient.secrets | indent 10 }}

command: ["bash"]
args:
- "-c"
- |
set -ex

VTCTLD_SVC=vtctld.{{ $namespace }}:15999
SECONDS=0
TIMEOUT_SECONDS=600
VTCTL_EXTRA_FLAGS=({{ include "format-flags-inline" $defaultVtctlclient.extraFlags }})

# poll every 5 seconds to see if keyspace is created
until vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC GetKeyspace {{ $keyspace.name }} > /dev/null 2>&1; do
if (( $SECONDS > $TIMEOUT_SECONDS )); then
echo "timed out waiting for keyspace {{ $keyspace.name }} to be ready"
exit 1
fi
sleep 5
done

vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC ApplyVSchema -vschema "$(cat <<END_OF_COMMAND
{{ $keyspace.vschema | indent 14 }}
END_OF_COMMAND
)" {{ $keyspace.name }}
volumes:
{{ include "user-secret-volumes" (.secrets | default $defaultVtctlclient.secrets) | indent 8 }}

{{- end -}}
{{- end -}}
{{- end -}}
116 changes: 116 additions & 0 deletions helm/vitess/templates/_shard.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
###################################
# shard initializations
###################################

{{ define "shard" -}}
{{- $cell := index . 0 -}}
{{- $keyspace := index . 1 -}}
{{- $shard := index . 2 -}}
{{- $defaultVtctlclient := index . 3 -}}
{{- $namespace := index . 4 -}}
{{- $totalTabletCount := index . 5 -}}

# sanitize inputs for labels
{{- $cellClean := include "clean-label" $cell.name -}}
{{- $keyspaceClean := include "clean-label" $keyspace.name -}}
{{- $shardClean := include "clean-label" $shard.name -}}
{{- $shardName := printf "%s-%s-%s" $cellClean $keyspaceClean $shardClean | lower -}}

{{- with $cell.vtctld }}
# define image to use
{{- $vitessTag := .vitessTag | default $defaultVtctlclient.vitessTag }}
---
###################################
# InitShardMaster Job
###################################
apiVersion: batch/v1
kind: Job
metadata:
name: {{ $shardName }}-init-shard-master
spec:
backoffLimit: 1
template:
spec:
restartPolicy: OnFailure
containers:
- name: init-shard-master
image: "vitess/vtctlclient:{{$vitessTag}}"
volumeMounts:
{{ include "user-secret-volumeMounts" $defaultVtctlclient.secrets | indent 10 }}

command: ["bash"]
args:
- "-c"
- |
set -ex

VTCTLD_SVC=vtctld.{{ $namespace }}:15999
SECONDS=0
TIMEOUT_SECONDS=600
VTCTL_EXTRA_FLAGS=({{ include "format-flags-inline" $defaultVtctlclient.extraFlags }})

# poll every 5 seconds to see if vtctld is ready
until vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC ListAllTablets {{ $cell.name }} > /dev/null 2>&1; do
if (( $SECONDS > $TIMEOUT_SECONDS )); then
echo "timed out waiting for vtctlclient to be ready"
exit 1
fi
sleep 5
done

until [ $TABLETS_READY ]; do
# get all the tablets in the current cell
cellTablets="$(vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC ListAllTablets {{ $cell.name }})"

# filter to only the tablets in our current shard
shardTablets=$( echo "$cellTablets" | awk 'substr( $5,1,{{ len $shardName }} ) == "{{ $shardName }}" {print $0}')

# check for a master tablet from the ListAllTablets call
masterTablet=$( echo "$shardTablets" | awk '$4 == "master" {print $1}')
if [ $masterTablet ]; then
echo "'$masterTablet' is already the master tablet, exiting without running InitShardMaster"
exit
fi

# check for a master tablet from the GetShard call
master_alias=$(vtctlclient ${VTLCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC GetShard {{ $keyspace.name }}/{{ $shard.name }} | jq '.master_alias.uid')
if [ "$master_alias" != "null" -a "$master_alias" != "" ]; then
echo "'$master_alias' is already the master tablet, exiting without running InitShardMaster"
exit
fi

# count the number of newlines for the given shard to get the tablet count
tabletCount=$( echo "$shardTablets" | wc | awk '{print $1}')

# check to see if the tablet count equals the expected tablet count
if [ $tabletCount == {{ $totalTabletCount }} ]; then
TABLETS_READY=true
else
if (( $SECONDS > $TIMEOUT_SECONDS )); then
echo "timed out waiting for tablets to be ready"
exit 1
fi

# wait 5 seconds for vttablets to continue getting ready
sleep 5
fi

done

# find the tablet id for the "-replica-0" stateful set for a given cell, keyspace and shard
tablet_id=$( echo "$shardTablets" | awk 'substr( $5,1,{{ add (len $shardName) 10 }} ) == "{{ $shardName }}-replica-0" {print $1}')

# initialize the shard master
until vtctlclient ${VTCTL_EXTRA_FLAGS[@]} -server $VTCTLD_SVC InitShardMaster -force {{ $keyspace.name }}/{{ $shard.name }} $tablet_id; do
if (( $SECONDS > $TIMEOUT_SECONDS )); then
echo "timed out waiting for InitShardMaster to succeed"
exit 1
fi
sleep 5
done
volumes:
{{ include "user-secret-volumes" (.secrets | default $defaultVtctlclient.secrets) | indent 8 }}


{{- end -}}
{{- end -}}
7 changes: 7 additions & 0 deletions helm/vitess/templates/_vtgate.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,10 @@ spec:
{{ include "vtgate-affinity" (tuple $cellClean $cell.region) | indent 6 }}

{{ if $cell.mysqlProtocol.enabled }}
{{ if eq $cell.mysqlProtocol.authType "secret" }}
initContainers:
{{ include "init-mysql-creds" (tuple $vitessTag $cell) | indent 8 }}
{{ end }}
{{ end }}

containers:
Expand Down Expand Up @@ -109,7 +111,12 @@ spec:
-grpc_port=15991
{{ if $cell.mysqlProtocol.enabled }}
-mysql_server_port=3306
{{ if eq $cell.mysqlProtocol.authType "secret" }}
-mysql_auth_server_impl="static"
-mysql_auth_server_static_file="/mysqlcreds/creds.json"
{{ else if eq $cell.mysqlProtocol.authType "none" }}
-mysql_auth_server_impl="none"
{{ end }}
{{ end }}
-service_map="grpc-vtgateservice"
-cells_to_watch={{$cell.name | quote}}
Expand Down
Loading