-
Notifications
You must be signed in to change notification settings - Fork 295
Avoid unnecessary node replacements when TLS bootstrapping is enabled #639
Avoid unnecessary node replacements when TLS bootstrapping is enabled #639
Conversation
The logic contained in this file will be re-used so we can also handle encryption of non-TLS assets.
Doing so, it allows us to re-use the same logic to handle encryption and caching of non-TLS resources.
Codecov Report
@@ Coverage Diff @@
## master #639 +/- ##
==========================================
- Coverage 38.26% 37.42% -0.85%
==========================================
Files 51 50 -1
Lines 3316 3177 -139
==========================================
- Hits 1269 1189 -80
+ Misses 1845 1812 -33
+ Partials 202 176 -26
Continue to review full report at Codecov.
|
This commit gets rid of the `token_config.go` file that were used to read/cache/fingerprint the auth tokens file, and instead re-use the same logic for reading/caching/fingerprinting TLS cert files. This commit also changes the way the auth tokens file is assembled; instead of storing the TLS bootstrap token in this file - and extracting/encrypting it in every update (which cause the issue reported in #633), we store the TLS bootstrap token separately in its own file (which are cached and fingerprinted like the other ones) and assemble the final auth tokens file in `cloud-config-controller`, during the asset decryption step.
This PR is ready for review. |
core/root/render/credentials.go
Outdated
@@ -25,9 +25,9 @@ func NewCredentialsRenderer(c *config.Cluster) CredentialsRenderer { | |||
} | |||
} | |||
|
|||
func (r credentialsRendererImpl) RenderTLSCerts(renderCredentialsOpts config.CredentialsOptions) error { | |||
func (r credentialsRendererImpl) RenderCerts(renderCredentialsOpts config.CredentialsOptions) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps RenderCredentials
should be a better name here as it generates not only certs but also tls-bootstrap token?
echo $(cat $authDir/tls-bootstrap.token),kubelet-bootstrap,10001,system:kubelet-bootstrap >> $authDir/computed-tokens.csv | ||
{{- end }} | ||
{{- if .AssetsConfig.HasAuthTokens }} | ||
cat $authDir/tokens.csv >> $authDir/computed-tokens.csv |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit but my preference is first write a work file like tokens.csv
to /var/run/coreos/tokens.csv
and keep the final one named $authDir/tokens.csv
rather than computed-tokens.csv
so that someone peeked into $authDir
finds only tokens.csv
and don't get confused like which one is the "final" tokens file passed to apiserver, computed-tokens.csv or tokens.csv?
@@ -1326,7 +1340,7 @@ write_files: | |||
- hostPath: | |||
path: /usr/share/ca-certificates | |||
name: ssl-certs-host | |||
{{if .AuthTokensConfig.HasTokens}} | |||
{{if .AssetsConfig.HasAnyAuthTokens}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW I like the function naming 👍
echo injecting token into the kubelet bootstrap kubeconfig file | ||
bootstrap_token=$(cat /etc/kubernetes/auth/kubelet-bootstrap.token); | ||
bootstrap_token=$(cat /etc/kubernetes/auth/tls-bootstrap.token); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just for my education but why did you rename it from kubelet- to tls-?
I believe you tried to match it against other relevant symbol names(TLSBootstrap, HasTLSBootstrapToken).
However, the naming in kubelet is kubelet-bootstrap
rather than tls-bootstrap
.
I guess you assumed (like me) that it is originally meant for kubelet-(tls-)bootstrap
hence named tls-bootstrap
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, my reasoning was to make it match with the symbols in Go code, and what this feature is called in upstream Kubernetes documentation, but I think I forgot to replace a few other names in templates.
What do you think it's the right thing to do here? Keep the old kublet-bootstrap
names throughout the templates, or replace them all by tls-bootstrap
(or even kubelet-tls-bootstrap
)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the clarification!
Yes. Renaming to tls-bootstrap makes more sense for me.
Also, I began to slightly prefer kubelet-tls-bootstrap because it produces no ambiguity even in the context of kube-aws. "kube-aws tls bootstrap" vs "kube-aws kubelet tls bootstrap" - the latter seems to better represent that it is for kubelet for me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
echo > $authDir/computed-tokens.csv | ||
|
||
{{- if .AssetsConfig.HasTLSBootstrapToken }} | ||
echo $(cat $authDir/tls-bootstrap.token),kubelet-bootstrap,10001,system:kubelet-bootstrap >> $authDir/computed-tokens.csv |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for being late in commenting something like this but
- my preference after read through code is, the token file named
tls-bootstrap-token
rather thantls-bootstrap.token
so that we can be consistent in the wordtoken
contained in basenames of both filestls-bootstrap-token
andtokens.csv
- It can be named like
tls-bootstra-token.txt
but I feels that it is too much 😃 and we already have various txt files without the.txt
extentions anyway
- It can be named like
@danielfm Thanks as always for your continuous efforts on this area 🙇 I've added several comments but LGTM overall. Anyway IMHO, what the most important thing in this PR is the code reduced by 500 lines! |
@mumoshu sure, I'll try to address these ASAP. |
The idea is to document that these files are temporary; their only purpose is to be used to generate the final $authDir/tokens.csv file.
@danielfm Thanks a lot for your efforts! LGTM 👍 |
* kubernetes-incubator/master: Fix "install-kube-system" script when "clusterAutoscaler" is disabled. Remove obsolete etcd locking logic Re: cluster-autoscaler support Make `go test` timeout longer enough for Travis Fixes kubernetes-retired#667 NVIDIA driver installation support on GPU instances (kubernetes-retired#645) Make kubelet flags more consistent Fix taint being assigned as labels Avoid unnecessary node replacements when TLS bootstrapping is enabled (kubernetes-retired#639) Update Kubernetes dashboard to v1.6.1. Update calico to v2.2.1. Fix typo in help message
…kubernetes-retired#639) Simplifies the code by re-using the same encryption/caching/fingerprinting logic both to cert files and TLS bootstrapping / auth tokens, and keeps the TLS bootstrap token in its own file instead of storing it directly in `./credentials/tokens.csv`. Concretely, instead of using the `./credentials/tokens.csv` file to store the TLS bootstrap token (and re-encrypting it every time, causing the bug in kubernetes-retired#633), we now store the bootstrap token in its own file, thus not causing unnecessary node pool replacement at every `kube-aws update`. TODO - [x] Test creation/update workflow with TLS bootstrapping disabled, without custom auth tokens - [x] Test creation/update workflow with TLS bootstrapping disabled, with custom auth tokens - [x] Test creation/update workflow with TLS bootstrapping enabled, without custom auth tokens - [x] Test creation/update workflow with TLS bootstrapping enabled, with custom auth tokens - [x] Address review comments and perform the last round of tests ## Actions Required If this get merged, we must not forget to inform users of the experimental TLS Bootstrap feature that the following actions are required: 1. Run `kube-aws render stack` to update controller/worker user data templates 2. Move the bootstrap token from `./credentials/tokens.csv` to its own file `./credentials/kubelet-tls-bootstrap-token` ``` # Remove the following line from ./credentials/tokens.csv, and move <token> # (with no leading/trailing blank chars and lines) to it's own file # ./credentials/kubelet-tls-bootstrap-token <token>,kubelet-bootstrap,10001,system:kubelet-bootstrap ``` 3. Run `kube-aws update` to update the cluster. This operation will cause controllers and workers to be replaced Fixes kubernetes-retired#633. Changelog * Rename files The logic contained in this file will be re-used so we can also handle encryption of non-TLS assets. * Rename functions and struct members to not mention TLS Doing so, it allows us to re-use the same logic to handle encryption and caching of non-TLS resources. * Reuse caching/encryption logic for the TLS bootstrap and auth tokens This commit gets rid of the `token_config.go` file that were used to read/cache/fingerprint the auth tokens file, and instead re-use the same logic for reading/caching/fingerprinting TLS cert files. This commit also changes the way the auth tokens file is assembled; instead of storing the TLS bootstrap token in this file - and extracting/encrypting it in every update (which cause the issue reported in kubernetes-retired#633), we store the TLS bootstrap token separately in its own file (which are cached and fingerprinted like the other ones) and assemble the final auth tokens file in `cloud-config-controller`, during the asset decryption step. * Avoid create/update cluster with TLS bootstrap enabled without token * Fix tests Ensures that tests that rely on the TLS bootstrapping feature run with a dummy bootstrap token in order to bypass validation. * Match conditional used in asset decryption script for consistency * Auto-generates the TLS bootstrap token if necessary * Rename function to a more appropriate name * Rename TLS token to make it clear it's related to the Kubelet * Suffix auth token and Kubelet TLS bootstrap files with `.tmp` The idea is to document that these files are temporary; their only purpose is to be used to generate the final $authDir/tokens.csv file.
Simplifies the code by re-using the same encryption/caching/fingerprinting logic both to cert files and TLS bootstrapping / auth tokens, and keeps the TLS bootstrap token in its own file instead of storing it directly in
./credentials/tokens.csv
.Concretely, instead of using the
./credentials/tokens.csv
file to store the TLS bootstrap token (and re-encrypting it every time, causing the bug in #633), we now store the bootstrap token in its own file, thus not causing unnecessary node pool replacement at everykube-aws update
.TODO
Actions Required
If this get merged, we must not forget to inform users of the experimental TLS Bootstrap feature that the following actions are required:
Run
kube-aws render stack
to update controller/worker user data templatesMove the bootstrap token from
./credentials/tokens.csv
to its own file./credentials/kubelet-tls-bootstrap-token
Run
kube-aws update
to update the cluster. This operation will cause controllers and workers to be replacedFixes #633.