add support for running agent helm chart on persistent volume #7123
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| authToken: auth-token | ||
| proxyAddr: proxy.example.com:3080 | ||
| kubeClusterName: test-kube-cluster-name | ||
| nodeSelector: | ||
| gravitational.io/k8s-role: node |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| authToken: auth-token | ||
| proxyAddr: proxy.example.com:3080 | ||
| kubeClusterName: test-kube-cluster-name | ||
| storage: | ||
| enabled: true | ||
| storageClassName: "aws-gp2" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| # | ||
| # Warning to maintainers, any changes to this file that are not specific to the StatefulSet need to also be duplicated | ||
| # in the deployment.yaml file. | ||
| # | ||
| {{- if .Values.storage.enabled }} | ||
| {{- if .Values.teleportVersionOverride }} | ||
| {{- $_ := set . "teleportVersion" .Values.teleportVersionOverride }} | ||
| {{- else }} | ||
| {{- $_ := set . "teleportVersion" .Chart.Version }} | ||
| {{- end }} | ||
| apiVersion: apps/v1 | ||
| kind: StatefulSet | ||
| metadata: | ||
| name: {{ .Release.Name }} | ||
| namespace: {{ .Release.Namespace }} | ||
| labels: | ||
| app: {{ .Release.Name }} | ||
| spec: | ||
| serviceName: {{ .Release.Name }} | ||
| replicas: {{ .Values.replicaCount }} | ||
| selector: | ||
| matchLabels: | ||
| app: {{ .Release.Name }} | ||
| template: | ||
| metadata: | ||
| annotations: | ||
| # ConfigMap checksum, to recreate the pod on config changes. | ||
| checksum/config: {{ include (print $.Template.BasePath "/config.yaml") . | sha256sum }} | ||
| {{- if .Values.annotations.pod }} | ||
| {{- toYaml .Values.annotations.pod | nindent 8 }} | ||
| {{- end }} | ||
| labels: | ||
| app: {{ .Release.Name }} | ||
| spec: | ||
| {{- if .Values.affinity }} | ||
| affinity: | ||
| {{- toYaml .Values.affinity | nindent 8 }} | ||
| {{- end }} | ||
| {{- if .Values.tolerations }} | ||
| tolerations: | ||
| {{- toYaml .Values.tolerations | nindent 6 }} | ||
| {{- end }} | ||
| {{- if .Values.initContainers }} | ||
| initContainers: {{- toYaml .Values.initContainers | nindent 6 }} | ||
| {{- if .Values.resources }} | ||
| resources: | ||
| {{- toYaml .Values.resources | nindent 10 }} | ||
| {{- end }} | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: | ||
| - all | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 9807 | ||
| volumeMounts: | ||
| - mountPath: /etc/teleport | ||
| name: "config" | ||
| readOnly: true | ||
| - mountPath: /etc/teleport-secrets | ||
| name: "auth-token" | ||
| readOnly: true | ||
| - mountPath: /var/lib/teleport | ||
| name: "data" | ||
| {{- if .Values.extraVolumeMounts }} | ||
| {{- toYaml .Values.extraVolumeMounts | nindent 8 }} | ||
| {{- end }} | ||
| {{- end }} | ||
| serviceAccountName: {{ .Values.serviceAccountName | default .Release.Name }} | ||
| {{- if .Values.nodeSelector }} | ||
| nodeSelector: | ||
| {{- toYaml .Values.nodeSelector | nindent 8 }} | ||
| {{- end }} | ||
| containers: | ||
| - name: "teleport" | ||
| image: "{{ if .Values.enterprise }}{{ .Values.enterpriseImage }}{{ else }}{{ .Values.image }}{{ end }}:{{ .teleportVersion }}" | ||
| args: | ||
| - "--diag-addr=0.0.0.0:3000" | ||
| {{- if .Values.insecureSkipProxyTLSVerify }} | ||
| - "--insecure" | ||
| {{- end }} | ||
| securityContext: | ||
| allowPrivilegeEscalation: false | ||
| capabilities: | ||
| drop: | ||
| - all | ||
| readOnlyRootFilesystem: true | ||
| runAsNonRoot: true | ||
| runAsUser: 9807 | ||
| ports: | ||
| - name: diag | ||
| containerPort: 3000 | ||
| protocol: TCP | ||
| livenessProbe: | ||
| httpGet: | ||
| path: /healthz | ||
| port: diag | ||
| initialDelaySeconds: 5 # wait 5s for agent to start | ||
| periodSeconds: 5 # poll health every 5s | ||
| failureThreshold: 6 # consider agent unhealthy after 30s (6 * 5s) | ||
| readinessProbe: | ||
| httpGet: | ||
| path: /readyz | ||
| port: diag | ||
| initialDelaySeconds: 5 # wait 5s for agent to register | ||
| periodSeconds: 5 # poll health every 5s | ||
| failureThreshold: 12 # consider agent unhealthy after 60s (12 * 5s) | ||
| {{- if .Values.resources }} | ||
| resources: | ||
| {{- toYaml .Values.resources | nindent 10 }} | ||
| {{- end }} | ||
| volumeMounts: | ||
| - mountPath: /etc/teleport | ||
| name: "config" | ||
| readOnly: true | ||
| - mountPath: /etc/teleport-secrets | ||
| name: "auth-token" | ||
| readOnly: true | ||
| - mountPath: /var/lib/teleport | ||
| name: "{{ .Release.Name }}-teleport-data" | ||
| {{- if .Values.extraVolumeMounts }} | ||
| {{- toYaml .Values.extraVolumeMounts | nindent 8 }} | ||
| {{- end }} | ||
| volumes: | ||
| - name: "config" | ||
| configMap: | ||
| name: {{ .Release.Name }} | ||
| - name: "auth-token" | ||
| secret: | ||
| secretName: {{ .Values.secretName }} | ||
| {{- if .Values.extraVolumes }} | ||
| {{- toYaml .Values.extraVolumes | nindent 6 }} | ||
| {{- end }} | ||
| volumeClaimTemplates: | ||
| - metadata: | ||
| name: "{{ .Release.Name }}-teleport-data" | ||
| spec: | ||
| accessModes: [ "ReadWriteOnce" ] | ||
| storageClassName: {{ .Values.storage.storageClassName }} | ||
| resources: | ||
| requests: | ||
| storage: {{ .Values.storage.requests }} | ||
| {{- end }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -57,6 +57,26 @@ podSecurityPolicy: | |
| # Labels is a map of key values pairs about this cluster | ||
| labels: {} | ||
|
|
||
| ################################################################ | ||
| # Values that must be provided if using persistent storage for Teleport state. | ||
| # | ||
| # Assigning a persistent volume to Teleport agent allows the agent to store its security association with the Teleport | ||
| # cluster for re-use when the pod is restarted. Without a persistent storage for this state, every time Teleport agent | ||
| # starts it must use the authToken to create a new registration with the cluster. By using the persistent volume the | ||
| # authToken can be routinely rotated without breaking agents' ability to restart, as the token is only used on the first | ||
| # startup. When persistent volumes are enabled, the agent will be deployed as a StatefulSet instead of a Deployment to | ||
| # Kubernetes. | ||
| # | ||
| # Fields: | ||
| # enabled: Set to true to enable the use of StatefulSets and Persistent volumes. | ||
| # storageClassName: The name of the kubernetes storage class to use when creating volumes. See https://kubernetes.io/docs/concepts/storage/storage-classes/ | ||
| # requests: The size of the volume to request from the persistent storage system | ||
| ################################################################ | ||
| storage: | ||
| enabled: false | ||
| storageClassName: "" | ||
| requests: 128Mi | ||
|
Contributor
There was a problem hiding this comment. I actually have no idea how big the teleport storage needs are.
Contributor
Author
There was a problem hiding this comment.
Last 7 days I've had this deployed / idle it's used 19MiB. I may have assumed incorrectly that because this is a node agent and logs/audit/etc are meant to be centralized that the default for this volume can be relatively small. Is there something I might be missing that would suggest a larger default may be needed here?
It depends on how the storage class is setup and presumably whether the block driver supports it. The storage class has a boolean field called allowVolumeExpansion, and if it's set, the volume can be expanded without data loss. It will interrupt the pod temporarily since the volume needs to be unmapped and the filesystem expanded (Edit: On AWS, IDK other systems might have an online expansion).
Contributor
There was a problem hiding this comment. I think this volume can be fairly small - no logs are really kept locally (they all go to Docker/containerd) and the sqlite database for the agent itself should only be holding a keypair and some cache data. |
||
|
|
||
| ################################################################ | ||
| # Values that you shouldn't need to change. | ||
| ################################################################ | ||
|
|
@@ -88,6 +108,10 @@ logLevel: INFO | |
| # https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity | ||
| affinity: {} | ||
|
|
||
| # nodeSelector to apply for pod assignment | ||
| # https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ | ||
| nodeSelector: {} | ||
|
Comment on lines
+111
to
+113
Contributor
There was a problem hiding this comment. Please add this to the correct section of https://github.com/gravitational/teleport/blob/master/docs/pages/kubernetes-access/helm/reference.mdx as well to keep everything documented. You can just copy a section and change the appropriate parts.
Contributor
Author
There was a problem hiding this comment. Alright, I added to the reference doc. |
||
|
|
||
| # Kubernetes annotations to apply | ||
| # https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ | ||
| annotations: | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.