tracking issue for kubeadm init phases #1163

neolit123 · 2018-10-08T17:02:35Z

this tracking issue goes outside of the scope for 1.13 AKA we have items for post 1.13.

this post will summarize the work that has to be done to graduate the kubeadm phases to beta (ga?).
the issue will track PRs, assignees, but also have extra actions that have to be handled.
if you wish to work on a certain item please add a comment about it.

discussion happened around this gdoc in the 17oct/2018 kubeadm office hours meeting:
https://docs.google.com/document/d/10ZQ5erWhab3GJH-CjCBT6NgvLJQygyqPkQUsx_y5snQ/edit?usp=sharing

notes:

we are keeping the kubeadm alpha command for now, but moving all viable phases in the bellow list to kubeadm init phase
some phases are going to be collapsed into a parent phase, but optionally sub-phases to be exposed over time.
some phases are shuffled around or removed.
we have plans to support aliases where a sub-phase can call the same code as another sub-phase.
some phases are going to be hidden which implies that they are going to be called by the backend in a specific order, but the user would not be able to call them from CLI or skip them.

list of phases (ordered)

preflight
assigned: @fabriziopandini
PR: Kubeadm graduate preflight phase kubernetes#69666
NOTE: this PR also serves as an example of how a new phases should be graduated.
kubelet-start
assigned: @fabriziopandini
PR: kubeadm - graduate kubelet-start phase kubernetes#70049

currently this phase has sub-phases, the idea is to collapse them into a single phase. shuffle folders around etc. this however removes the control if the user wants to write a kubelet config, modify it and then start. so we might want to expose it eventually based on user feedback.

certs
assigned: @ereslibre
PR: kubeadm: Graduate certs phase kubernetes#70331

should be a straight forward one, move all cert sub commands as sub phases.
renew needs to not be a sub-phase of certs, extra item bellow.

kubeconfig
assigned: @fabriziopandini
PR: Kubeadm graduate kubeconfig phase kubernetes#70098

same as certs,
user however should be kept as an utility under kubeadm alpha kubeconfig user and not being a phase.

control-plane
assigned: @neolit123
PR: kubeadm: graduate the control-plane phase kubernetes#70097

contains sub-phases for the control plane components. under api-server we also need to handle audit-policy (currently feature gated as a separate phase). there are plans to add aliases for certs here and eventually also have sub-sub-commands under each control plane component to 1) write config 2) write cert

etcd
assigned: @neolit123
PR: kubeadm: graduate etcd phase kubernetes#70388

similar to control-plane, optionally enable 1) write config 2) write cert as sub phases here eventually too.

wait-control-plane
assigned: @neolit123
PR: kubeadm: graduate wait-control-plane phase kubernetes#70512

hidden phase!

upload-config
assigned: @yagonobre
PR: kubeadm graduate upload-config phase kubernetes#70405

this should including uploading the kubelet config (currently under kubelet phase) and the kubeadm config. both should be exposed as sub-phases.

annotate-cri-socket should be part of uploading the kubelet config

mark-control-plane
assigned: @RA489 @neolit123
PR: kubeadm: graduate the mark-control-plane-phase kubernetes#70885

renamed from mark-master.

bootstrap-token
assigned: @yagonobre
PR: kubeadm graduate bootstrap-token phase kubernetes#70727

should contain all of these, but collapsed and not exposed as sub-phases.

create (the function is CreateOrUpdate...)
allow-post-rbac-csr
auto-approve-node-tokens
auto-approve-node-cert-rotation
create-bootstrap-config-map 
create-cluster-info-rbac

addons
assigned: @yagonobre
PR: Kubeadm graduate addons phase kubernetes#70354

should contain dns and proxy as subphases exposed in that order.

other action items:

printing the join command should happen after all the phases in init have finished
assigned: @yuexiao-wang
PR: kubeadm: print the join command kubernetes#70795

this needs to handle the case where the creation of a token was skipped.

alpha phase certs renew should be moved to alpha certs renew (not a phase)
assigned: @fabriziopandini
PR: Kubeadm alpha cmd kubernetes#70333
app/phases/upgrade needs to be moved to generic kubeadm utils (not a phase)
assigned: @yuexiao-wang
PR: kubeadm: move app/phases/upgrade to generic kubeadm utils kubernetes#70346
certs/pkiutil needs to be moved to generic kubeadm utils (not a phase)
assigned: @yuexiao-wang
PR: kubeadm: move certs/pkiutil to generic kubeadm utils kubernetes#70336
bootstraptoken/node/token.go the utils in here need to be moved outside of k/k
assigned: @neolit123
PR: kubeadm: remove TODO about moving token utils to client-go kubernetes#69979
enable-dynamic-kubelet-config should be moved outside of phases
assigned: @yagonobre
PR: Remove DynamicKubeletConfig from init and join workflow kubernetes#70849

it's feature gated.

to be decided....
the new home for this should be "k8s.io/cluster-bootstrap/token/util"
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cluster-bootstrap/token/util/helpers.go

The text was updated successfully, but these errors were encountered:

fabriziopandini · 2018-10-10T14:23:39Z

fyi
This was the list of phases proposed some time ago

  preflight                  Run master pre-flight checks
  certs|certificates         Generates all PKI assets necessary to establish the control plane
    /ca                        Generates a self-signed kubernetes CA to provision identities for components of the cluster
    /apiserver                 Generates an API server serving certificate and key
    /apiserver-kubelet-client  Generates a client certificate for the API server to connect to the kubelets securely
    /etcd-ca                   Generates a self-signed CA to provision identities for etcd
    /etcd-server               Generates an etcd serving certificate and key
    /etcd-peer                 Generates an etcd peer certificate and key
    /etcd-healthcheck-client   Generates a client certificate for liveness probes to healthcheck etcd
    /apiserver-etcd-client     Generates a client certificate for the API server to connect to etcd securely
    /sa                        Generates a private key for signing service account tokens along with its public key
    /front-proxy-ca            Generates a front proxy CA certificate and key for a Kubernetes cluster
    /front-proxy-client        Generates a front proxy CA client certificate and key for a Kubernetes cluster
  kubeconfig                 Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
    /admin                     Generates a kubeconfig file for the admin to use and for kubeadm itself
    /kubelet                   Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes.
    /controller-manager        Generates a kubeconfig file for the controller manager to use
    /scheduler                 Generates a kubeconfig file for the scheduler to use
  audit-policy               Generates the audit-policy to use in the API server configuration (featureGated)
  controlplane               Generates all static Pod manifest files necessary to establish the control plane
    /apiserver                 Generates the API server static Pod manifest.
    /controller-manager        Generates the controller-manager static Pod manifest.
    /scheduler                 Generates the scheduler static Pod manifest.
  etcd                       Generates the static Pod manifest file for a local, single-node etcd instance
  init-kubeletconfig         (featureGated) Writes base configuration of kubelets to disk
  upload-kubeletconfig       (featureGated) Uploads kubelet config to a configMap and use it as a configSource for this node
  upload-config              Uploads the currently used configuration for kubeadm to a ConfigMap
  mark-master                Marks a node as master
  bootstrap-token            Makes all the bootstrap token configurations and creates an initial token
    /token                     Creates an initial bootstrap token to be used for node joining
    /cluster-info|clusterinfo  Uploads the cluster-info ConfigMap from the given kubeconfig file
    /allow-post-csr            Configures RBAC to allow node bootstrap tokens to post CSR in order for nodes to get long term certificate credentials
    /allow-auto-approve        Configures RBAC rules to allow the CSR approver controller automatically approve CSR from a node bootstrap token
  addons                     Installs required addons for passing Conformance tests
    /dns                       Installs the dns addon to a Kubernetes cluster
    /proxy                     Installs the proxy addon to a Kubernetes cluster
  selfhosting|self-hosting   Makes a kubeadm cluster self-hosted (featureGated)

timothysc · 2018-10-11T15:42:03Z

This looks good, but I think we might want to change some of the names, and obviously move selfhosting into it's own space.

fabriziopandini · 2018-10-16T11:14:22Z

@neolit123 could you kindly change this issues into a tracking issue /help-wanted that we can use to assign part of this effort to volunteers...

I think that @ereslibre, @RA489, @Klaven can help on this effort once the PR with the first phase implementation merges; ideally who takes in charge a phase should implement related subphases as well

neolit123 · 2018-10-16T11:17:43Z

@fabriziopandini
done.

neolit123 · 2018-10-16T11:18:36Z

are we going to discuss the list of phases this weeks kubeadm office hour? we need a solid list...

ereslibre · 2018-10-16T11:27:12Z

I won't be able to join tomorrows kubeadm office hours :(, but count me in for any ongoing effort!

neolit123 · 2018-10-17T17:15:59Z

now that we have the list of phases i will update this tracking issue tomorrow.

fabriziopandini · 2018-10-18T19:38:20Z

@neolit123 thanks for this write-up. It is great starting point
@ereslibre, @RA489, @Klaven time for volunteering!

neolit123 · 2018-10-18T19:50:21Z

a couple of requests for volunteers :)

please, don't mention this issue in commit messages, because it will spam it on commit rebases.
in PR descriptions use, Ref #this-issue-num instead of Fixes #this-issue-num, so that we don't close it on merge.

thanks.

ereslibre · 2018-10-19T08:21:42Z

For starting I can take the certs one!

neolit123 · 2018-10-19T14:06:18Z

@fabriziopandini
btw i'm not exactly sure how we can sync the work here, unless phases are implemented in the presented order. the code in runInit() would still have to be present and we have to hold PRs that implement say mark-control-plane in the new fashion. otherwise we are going to break master.

fabriziopandini · 2018-10-19T14:34:28Z

@neolit123 I'm confident we can sync via slack/this issue

BTW, I'm taking phase kubelet-start

neolit123 · 2018-10-19T20:26:36Z

@fabriziopandini ok, sounds good.

kubelet-start has a bit of a blocker in terms of the "stop" part as per @chuckha 's PR here:
kubernetes/kubernetes#69985
we need to discuss what to do with that.

fabriziopandini · 2018-10-22T11:51:34Z

taking kubeconfig

neolit123 · 2018-10-22T12:36:46Z

taking control-plane but really want the PR that removes placeholder docs merged...

yagonobre · 2018-10-29T03:03:13Z

alpha phase certs renew was done in kubernetes/kubernetes#70333, I can take bootstrap-token and addons.

bart0sh · 2018-10-29T21:16:49Z

I can take this: "app/phases/upgrade needs to be moved to generic kubeadm utils (not a phase)"

yagonobre · 2018-10-29T21:25:14Z

@bart0sh this was done on kubernetes/kubernetes#70346

bart0sh · 2018-10-29T21:30:34Z

@yagonobre thanks for pointing it out. won't take it then. I thought it's not assigned:

 app/phases/upgrade needs to be moved to generic kubeadm utils (not a phase)

assigned: TODO

neolit123 · 2018-10-29T21:32:09Z

sorry, just updated it.

yagonobre · 2018-10-30T03:38:13Z

What is the expected subphases on bootstrap-token? I'm taking upload-config in the meantime.

yagonobre · 2018-10-30T04:22:29Z

taking enable-dynamic-kubelet-config

neolit123 · 2018-10-30T14:23:43Z

@yagonobre

What is the expected subphases on bootstrap-token? I'm taking upload-config in the meantime.

as commented there, the listed sub-phases should be just collapsed in a single phase.

should contain all of these, but collapsed and not exposed as sub-phases.

yagonobre · 2018-10-30T14:29:31Z

ok, taking bootstrap-token again

neolit123 · 2018-10-30T14:44:47Z

PSA, the phases need to be merged in the above order as we don't want to break e2e tests.

fabriziopandini · 2018-10-30T18:06:18Z

@neolit123 annotateCRI IMO doesn't deserve a first level phase because it ideally should go away as soon as possible.. What about having annotateCRI as part of the upload kubelet config subphases? (Run Upload kubelet config writes the config map and annotates CRI)
@yagonobre wdyt?

neolit123 · 2018-10-30T18:07:38Z

@fabriziopandini
SGTM, i will update the list.
responded here too: kubernetes/kubernetes#70405 (comment)

neolit123 · 2018-10-31T21:03:09Z

@RA489 hi, do you have a PR in the works for mark-control-plane?
we might be able to merge it this week if so.

RA489 · 2018-11-01T04:29:52Z

@neolit123 WIP

neolit123 · 2018-11-13T20:05:34Z

the work here is done!

the only remaining items are these, but we can track then individually:
kubernetes/kubernetes#70346
kubernetes/kubernetes#69979

we did a pretty nice organization here and managed to tackle the list of tasks pretty well!!
thank you @fabriziopandini @yagonobre @yuexiao-wang @ereslibre @RA489

i will also send a shoutout to everyone for the community meeting on Thursday.

neolit123 added the kind/design Categorizes issue or PR as related to design. label Oct 8, 2018

neolit123 assigned neolit123 and fabriziopandini Oct 8, 2018

neolit123 added this to the v1.13 milestone Oct 8, 2018

fabriziopandini mentioned this issue Oct 10, 2018

kubeadm refactor cmd init kubernetes/kubernetes#69622

Merged

fabriziopandini mentioned this issue Oct 11, 2018

Kubeadm graduate preflight phase kubernetes/kubernetes#69666

Merged

1 task

fabriziopandini mentioned this issue Oct 11, 2018

kubeadm add phase runner kubernetes/kubernetes#69684

Merged

neolit123 added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/tracking-issue labels Oct 16, 2018

neolit123 changed the title ~~discuss potential list of phases in 1.13~~ tracking issue for kubeadm init phases in 1.13 Oct 16, 2018

neolit123 changed the title ~~tracking issue for kubeadm init phases in 1.13~~ tracking issue for kubeadm init phases Oct 18, 2018

neolit123 mentioned this issue Oct 18, 2018

kubeadm: remove TODO about moving token utils to client-go kubernetes/kubernetes#69979

Closed

fabriziopandini mentioned this issue Oct 20, 2018

kubeadm - graduate kubelet-start phase kubernetes/kubernetes#70049

Merged

neolit123 mentioned this issue Oct 22, 2018

kubeadm: graduate the control-plane phase kubernetes/kubernetes#70097

Merged

fabriziopandini mentioned this issue Oct 22, 2018

Kubeadm graduate kubeconfig phase kubernetes/kubernetes#70098

Merged

yagonobre mentioned this issue Oct 29, 2018

Kubeadm graduate addons phase kubernetes/kubernetes#70354

Merged

neolit123 mentioned this issue Oct 29, 2018

kubeadm: graduate etcd phase kubernetes/kubernetes#70388

Merged

yagonobre mentioned this issue Oct 30, 2018

kubeadm graduate upload-config phase kubernetes/kubernetes#70405

Merged

yagonobre mentioned this issue Oct 30, 2018

[WIP] kubeadm graduate enable-dynamic-kubelet-config phase kubernetes/kubernetes#70410

Closed

yagonobre mentioned this issue Oct 30, 2018

Add Aliases to workflow phases kubernetes/kubernetes#70444

Closed

neolit123 mentioned this issue Nov 1, 2018

kubeadm: graduate wait-control-plane phase kubernetes/kubernetes#70512

Merged

RA489 mentioned this issue Nov 4, 2018

kubeadm: mark-control-plane phase kubernetes/kubernetes#70625

Closed

neolit123 added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label Nov 5, 2018

yagonobre mentioned this issue Nov 7, 2018

kubeadm graduate bootstrap-token phase kubernetes/kubernetes#70727

Merged

yuexiao-wang mentioned this issue Nov 8, 2018

kubeadm: print the join command kubernetes/kubernetes#70795

Merged

This was referenced Nov 8, 2018

Dramatically Simplify Kubernetes Cluster Creation (kubeadm umbrella issue). kubernetes/enhancements#11

Closed

kubeadm: graduate the mark-control-plane-phase kubernetes/kubernetes#70885

Merged

neolit123 closed this as completed Nov 13, 2018

ereslibre mentioned this issue Jan 27, 2019

REQUEST: New membership for ereslibre kubernetes/org#421

Closed

6 tasks

neolit123 mentioned this issue Sep 15, 2020

Kind fails kubadm command: unknown flag --skip-phases kubernetes-sigs/kind#1853

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tracking issue for kubeadm init phases #1163

tracking issue for kubeadm init phases #1163

neolit123 commented Oct 8, 2018 •

edited

Loading

fabriziopandini commented Oct 10, 2018 •

edited by neolit123

Loading

timothysc commented Oct 11, 2018

fabriziopandini commented Oct 16, 2018

neolit123 commented Oct 16, 2018

neolit123 commented Oct 16, 2018 •

edited

Loading

ereslibre commented Oct 16, 2018

neolit123 commented Oct 17, 2018

fabriziopandini commented Oct 18, 2018

neolit123 commented Oct 18, 2018 •

edited

Loading

ereslibre commented Oct 19, 2018

neolit123 commented Oct 19, 2018

fabriziopandini commented Oct 19, 2018

neolit123 commented Oct 19, 2018

fabriziopandini commented Oct 22, 2018

neolit123 commented Oct 22, 2018

yagonobre commented Oct 29, 2018

bart0sh commented Oct 29, 2018

yagonobre commented Oct 29, 2018

bart0sh commented Oct 29, 2018

neolit123 commented Oct 29, 2018

yagonobre commented Oct 30, 2018

yagonobre commented Oct 30, 2018

neolit123 commented Oct 30, 2018

yagonobre commented Oct 30, 2018

neolit123 commented Oct 30, 2018 •

edited

Loading

fabriziopandini commented Oct 30, 2018

neolit123 commented Oct 30, 2018 •

edited

Loading

neolit123 commented Oct 31, 2018

RA489 commented Nov 1, 2018

neolit123 commented Nov 13, 2018

tracking issue for kubeadm init phases #1163

tracking issue for kubeadm init phases #1163

Comments

neolit123 commented Oct 8, 2018 • edited Loading

list of phases (ordered)

other action items:

fabriziopandini commented Oct 10, 2018 • edited by neolit123 Loading

timothysc commented Oct 11, 2018

fabriziopandini commented Oct 16, 2018

neolit123 commented Oct 16, 2018

neolit123 commented Oct 16, 2018 • edited Loading

ereslibre commented Oct 16, 2018

neolit123 commented Oct 17, 2018

fabriziopandini commented Oct 18, 2018

neolit123 commented Oct 18, 2018 • edited Loading

ereslibre commented Oct 19, 2018

neolit123 commented Oct 19, 2018

fabriziopandini commented Oct 19, 2018

neolit123 commented Oct 19, 2018

fabriziopandini commented Oct 22, 2018

neolit123 commented Oct 22, 2018

yagonobre commented Oct 29, 2018

bart0sh commented Oct 29, 2018

yagonobre commented Oct 29, 2018

bart0sh commented Oct 29, 2018

neolit123 commented Oct 29, 2018

yagonobre commented Oct 30, 2018

yagonobre commented Oct 30, 2018

neolit123 commented Oct 30, 2018

yagonobre commented Oct 30, 2018

neolit123 commented Oct 30, 2018 • edited Loading

fabriziopandini commented Oct 30, 2018

neolit123 commented Oct 30, 2018 • edited Loading

neolit123 commented Oct 31, 2018

RA489 commented Nov 1, 2018

neolit123 commented Nov 13, 2018

neolit123 commented Oct 8, 2018 •

edited

Loading

fabriziopandini commented Oct 10, 2018 •

edited by neolit123

Loading

neolit123 commented Oct 16, 2018 •

edited

Loading

neolit123 commented Oct 18, 2018 •

edited

Loading

neolit123 commented Oct 30, 2018 •

edited

Loading

neolit123 commented Oct 30, 2018 •

edited

Loading