[WIP] *: add logic for cluster-etcd-operator toggle #2608

hexfusion · 2019-11-01T14:59:57Z

NOTE cluster-etcd-operator is out for 4.3

In 4.3 cluster-etcd-operator will take over the process of bootstrapping the etcd cluster. To provide a clear path to disable/revert these changes we have setup the following conditional logic.

MCO: The MCO render command invoked in bootkube has a new optional flag to pass the value of the cluster-etcd-operator image[1]. The availability of this flags value[2] is used to conditionally adjust the etcd-member static pod spec allowing it to use the new bootstrapping method via the operator or fall back to the 4.2 SRV method.

Installer: The installer in 4.3 has a few notable changed introduced by this PR. First of all the render command populates a static pod manifest which creates a single member etcd cluster. After we have the single node cluster we can progress and cluster-operator can be deployed. This speeds up the time it takes for the operators to begin to reconcile as we are no longer waiting for all 3 etcds to bootstrap before we progress the operators.

cluster-etcd-operator: CEO is currently set as Unmanaged[3]. This allows us to include the CEO in CVO operator payload while setting the controllers to perform noop. This short term phase allows us to merge this PR proving that we can at the same time have CEO included in CVO but still use the old SRV bootstrap.

Revert Plan: If a case existed where we had a design error and the operator needed to be pulled from 4.3.
:

[1] https://github.com/openshift/installer/pull/2608/files#diff-ce82c1d8a44f7dfc41dfc024085ccfeeR298
[2] https://github.com/openshift/machine-config-operator/blob/bd846958bc95d049547164046a962054fca093df/templates/master/00-master/_base/files/etc-kubernetes-manifests-etcd-member.yaml#L22
[3] https://github.com/openshift/cluster-etcd-operator/blob/master/manifests/0000_12_etcd-operator_01_operator.cr.yaml#L8

Depends on:

hexfusion · 2019-11-01T15:28:00Z

/test e2e-aws

cmd/openshift-install/create.go

hexfusion · 2019-11-01T16:16:54Z

adding DNS for other cloud providers

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

pkg/asset/manifests/operators.go

patrickdillon · 2019-11-01T19:43:26Z

~~In general, installer repo expects commit messages. See formatting here: https://github.com/patrickdillon/installer/blob/master/CONTRIBUTING.md#commit-message-format~~

Just reread your PR:

I will write up a full changelog soon.

hexfusion · 2019-11-01T20:11:41Z

@patrickdillon I will clean this all up before we get to merging time.

hexfusion · 2019-11-01T20:20:56Z

/cc @abhinavdahiya @crawford

Curious about general thoughts. The idea for this PR is to set CEO up for full testing. But we are testing with CEO disabled which would be the result if we had to revert.

abhinavdahiya · 2019-11-01T21:27:07Z

data/data/aws/route53/base.tf

  records = [var.etcd_ip_addresses[count.index]]
 }

+resource "aws_route53_record" "bootstrap_a_node" {


nit, we try to keep all the bootstrap resources ie resources that are used by bootstrap-host and should be deleted when bootstrapping is complete, in the bootstrap module.

So I think a better way would be to pass the internal_zone_id, cluster_domain to the bootstrap module and create the dns record there??

We would like to resolve this in a patch after this PR if possible.

We would like to resolve this in a patch after this PR if possible.

hmm, this shouldn't be too much change..
just want to make sure we don't delete resources that aren't meant to be deleted..

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

hexfusion · 2019-11-05T02:07:18Z

/retest

hexfusion · 2019-11-05T04:05:14Z

/skip

data/data/aws/route53/base.tf

abhinavdahiya · 2019-11-25T22:23:30Z

data/data/aws/route53/base.tf

["${var.bootstrap_ip_address}"] -> var.bootstrap_ip_address

["${var.bootstrap_ip_address}"] -> [var.bootstrap_ip_address] didn't realize that the records needed a list.

abhinavdahiya · 2019-11-25T22:25:36Z

data/data/azure/dns/dns.tf

https://github.com/openshift/installer/pull/2608/files#diff-1dd6be44797de18d5c235fc324f8ee2aR92

i think this should be something like https://github.com/openshift/installer/pull/2608/files#diff-764adb23dcc0edbbebc09192eb233e9aR3

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

abhinavdahiya · 2019-11-25T22:29:40Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

not sure why you needs these volumes?

abhinavdahiya · 2019-11-25T22:31:45Z

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

why do we start the etcd member here?
https://github.com/openshift/installer/pull/2608/files#diff-ce82c1d8a44f7dfc41dfc024085ccfeeR200
but the bootstrap-apiserver puts the manifests to be run when bootstrapping the cluster..

Is it because we think the lifecycle of that pod is separate from the bootstrap-control-plane

hexfusion · 2019-11-26T12:08:54Z

/retest

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template

Signed-off-by: Sam Batschelet <[email protected]>

hexfusion · 2019-11-26T14:47:13Z

FTR will rebase this into something reasonable when I am done with the review.

openshift-ci-robot · 2019-11-26T14:58:16Z

@hexfusion: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`b7e62e4`	link	`/test e2e-aws-scaleup-rhel7`
ci/prow/e2e-aws	`b7e62e4`	link	`/test e2e-aws`
ci/prow/e2e-aws-fips	`b7e62e4`	link	`/test e2e-aws-fips`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hexfusion · 2019-11-27T22:42:13Z

We have reworked this into a simpler solution, continued here #2730

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 1, 2019

openshift-ci-robot requested review from jstuever and patrickdillon November 1, 2019 15:00

hexfusion commented Nov 1, 2019

View reviewed changes

cmd/openshift-install/create.go Outdated Show resolved Hide resolved

cmd/openshift-install/create.go Outdated Show resolved Hide resolved

hexfusion force-pushed the 4.3-cluster-etcd-operator-v1 branch 4 times, most recently from ffd10f8 to 58545f4 Compare November 1, 2019 18:20

hexfusion commented Nov 1, 2019

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

patrickdillon reviewed Nov 1, 2019

View reviewed changes

pkg/asset/manifests/operators.go Outdated Show resolved Hide resolved

hexfusion force-pushed the 4.3-cluster-etcd-operator-v1 branch from 2a4900a to a7e55c1 Compare November 1, 2019 20:01

openshift-ci-robot requested review from abhinavdahiya and crawford November 1, 2019 20:20

abhinavdahiya reviewed Nov 1, 2019

View reviewed changes

hexfusion changed the title ~~WIP *: add logic for cluster-etcd-operator toggle~~ *: add logic for cluster-etcd-operator toggle Nov 4, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2019

hexfusion changed the title *: add logic for cluster-etcd-operator toggle WIP *: add logic for cluster-etcd-operator toggle Nov 4, 2019

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 4, 2019

alaypatel07 reviewed Nov 4, 2019

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

hexfusion changed the title ~~WIP *: add logic for cluster-etcd-operator toggle~~ *: add logic for cluster-etcd-operator toggle Nov 5, 2019

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 5, 2019

openshift-ci-robot assigned alaypatel07 Nov 5, 2019

abhinavdahiya reviewed Nov 25, 2019

View reviewed changes

data/data/aws/route53/base.tf Outdated Show resolved Hide resolved

abhinavdahiya reviewed Nov 25, 2019

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

abhinavdahiya reviewed Nov 25, 2019

View reviewed changes

hexfusion force-pushed the 4.3-cluster-etcd-operator-v1 branch 3 times, most recently from f46adcc to d364191 Compare November 26, 2019 11:02

hexfusion force-pushed the 4.3-cluster-etcd-operator-v1 branch from d364191 to 3158e81 Compare November 26, 2019 14:22

abhinavdahiya reviewed Nov 26, 2019

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

abhinavdahiya reviewed Nov 26, 2019

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

abhinavdahiya reviewed Nov 26, 2019

View reviewed changes

data/data/bootstrap/files/usr/local/bin/bootkube.sh.template Outdated Show resolved Hide resolved

jeckersb and others added 11 commits November 26, 2019 09:35

bootstrap: use bootstrap etcd

0d555e6

etcd: populate etcd-bootstrap records

1f0edf2

Signed-off-by: Sam Batschelet <[email protected]>

Move kube client agent commands to CEO

cc34775

Add cluster-etcd-operator to bootkube template

cf3780f

operators.go: hard code etcd endpoints to bootstrap node and 3 masters

6153fe7

bootkube.sh.template: render CEO only if present in release payload

09f6648

data/data: add bootstrap A records for GCP and Azure

97f323e

Signed-off-by: Sam Batschelet <[email protected]>

bootkube.sh.template: copy etcd-bootstrap manifest only if CEO enabled

36db646

bootkube.sh.template: handle Managed/Unmanaged operator state

aca5625

Signed-off-by: Sam Batschelet <[email protected]>

bootkube.sh: improve readability

f558ed4

Signed-off-by: Sam Batschelet <[email protected]>

bootkube: restructure ceo blocks

b7e62e4

Signed-off-by: Sam Batschelet <[email protected]>

hexfusion force-pushed the 4.3-cluster-etcd-operator-v1 branch from 3158e81 to b7e62e4 Compare November 26, 2019 14:40

hexfusion closed this Nov 27, 2019

[WIP] *: add logic for cluster-etcd-operator toggle #2608

[WIP] *: add logic for cluster-etcd-operator toggle #2608

Uh oh!

Conversation

hexfusion commented Nov 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hexfusion commented Nov 1, 2019

Uh oh!

Uh oh!

Uh oh!

hexfusion commented Nov 1, 2019

Uh oh!

Uh oh!

Uh oh!

patrickdillon commented Nov 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hexfusion commented Nov 1, 2019

Uh oh!

hexfusion commented Nov 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hexfusion commented Nov 5, 2019

Uh oh!

hexfusion commented Nov 5, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhinavdahiya Nov 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hexfusion commented Nov 26, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hexfusion commented Nov 26, 2019

Uh oh!

openshift-ci-robot commented Nov 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hexfusion commented Nov 27, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

hexfusion commented Nov 1, 2019 •

edited

Loading

patrickdillon commented Nov 1, 2019 •

edited

Loading

hexfusion commented Nov 1, 2019 •

edited

Loading

abhinavdahiya Nov 25, 2019 •

edited

Loading

openshift-ci-robot commented Nov 26, 2019 •

edited

Loading