openstack: Add coredns, mdns-publisher and haproxy static pods #740

trown · 2019-05-13T19:02:11Z

This is part of the work to remove the "service" VM from the OpenStack architecture. Instead of running coredns on this extra VM, we will run it in a static pod on all masters. We are using the mdns work from metal3.io in order to coordinate the DNS records on the masters.

cgwalters

/approve

cgwalters · 2019-05-14T13:48:45Z

templates/master/00-master/openstack/files/openstack-mdns-publisher.yaml

This will need to be added to image-references and come from the payload.

cgwalters · 2019-05-14T13:48:52Z

templates/master/00-master/openstack/files/openstack-mdns-publisher.yaml

cgwalters · 2019-05-14T13:50:09Z

templates/master/00-master/openstack/files/openstack-mdns-publisher.yaml

I think the more correct approach is for the image to install the Python it wants.

cgwalters · 2019-05-14T13:51:32Z

templates/master/00-master/openstack/files/openstack-mdns-publisher.yaml

This feels a bit hacky...

I agree. I added a TODO to come up with a better method of getting these.

trown · 2019-05-15T15:50:29Z

/hold

need to properly publish the coredns and mdns-publisher images this patch uses

abhinavdahiya · 2019-05-16T04:31:36Z

templates/master/00-master/openstack/files/openstack-coredns.yaml

nit: remove kni

abhinavdahiya · 2019-05-16T04:32:47Z

templates/master/00-master/openstack/files/openstack-coredns.yaml

These requests are pretty high.

This is part of the work to remove the service VM from the openstack architecture. This relies on the coredns/mdns static pods setup in: openshift/machine-config-operator/pull/740

openshift#740 and openshift/installer#1959 depend on each other. We need to break this dependency to allow them to merge while keeping the e2e-openstack job green. This patch disables the static pods added for the openstack platform in openshift#740 unless the installer provided the needed info set via openshift/installer#1959. It can be safely reverted once openshift/installer#1959 merges.

… pods In order to have a more fault tolerant networking architecture, we are replacing the functionality of the service vm with a number of static pod resources that are run on master and worker nodes.

We use Baremetal-RuntimeCfg to clean up our code, as well as align our architecture much more closely with that of the Baremetal Team.

We need the wildcard record on the master nodes to resolve the Ingress IP. Since the `hosts` plugin doesn't support wildcards, it is replaced with the RFC 1035-style zone DB file which serves the API and wildcard records via the `file` plugin. This also adds the API record to both the bootstrap and master nodes.

…tches openshift#740 and openshift/installer#1959 depend on each other. We need to break this dependency to allow them to merge while keeping the e2e-openstack job green. This patch disables the static pods added for the openstack platform in openshift#740 unless the installer provided the needed info set via openshift/installer#1959. It can be safely reverted once openshift/installer#1959 merges.

mandre · 2019-07-31T06:50:40Z

/test e2e-openstack

mandre · 2019-07-31T08:58:57Z

/test e2e-openstack

tomassedovic · 2019-07-31T09:49:01Z

/retest

tomassedovic · 2019-07-31T10:42:00Z

/retest

This worked for me manually on the master. Looks more like a timeout on the openstack job.

tomassedovic · 2019-07-31T12:15:36Z

/retest

tomassedovic · 2019-08-01T11:33:18Z

@cgwalters @runcom would you mind having a look at this PR?

It's the OpenStack equivalent to: #795

We've tested it locally (with and without the corresponding installer PR openshift/installer#1959), it passes the CI and OpenStack has a FFE for this work. Is there anything else you'd like us to do?

cgwalters · 2019-08-01T21:18:07Z

pkg/controller/template/render.go

 	// Loop over templates/common which applies everywhere
 	for _, dir := range []string{platformBase, platform} {
+		// Bypass OpenStack template rendering until
+		// https://github.com/openshift/installer/pull/1959 merges


Hum...so if I understand this correctly, this PR is adding all of the code, and then we need to get the installer PR in first, then we'll do a PR to drop these conditionals?

Yes that's the intention. The installer PR depends on the changes there, but this PR depends on the changes in the installer PR so this let's us break that cycle.

The conditionals are isolated in a single commit we can revert afterwards.

cgwalters · 2019-08-01T21:18:30Z

/lgtm

openshift-ci-robot · 2019-08-01T21:18:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, trown

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cgwalters · 2019-08-01T21:19:38Z

I find this part encouraging:

These differences are not fundamental to OpenStack and we will be
looking at aligning more closely with the Baremetal provider in the
future.

Bigger picture, I wonder whether we should consider having the cluster own DNS by default across the board in a uniform way. That's another topic though.

The commit message in openshift/installer#1959 is great, very helpful!

openshift-bot · 2019-08-01T23:50:55Z

/retest

Please review the full test history for this PR and help us cut down flakes.

The experimental OpenStack backend used to create an extra server running DNS and load balancer services that the cluster needed. OpenStack does not always come with DNSaaS or LBaaS so we had to provide the functionality the OpenShift cluster depends on (e.g. the etcd SRV records, the api-int records & load balancing, etc.). This approach is undesirable for two reasons: first, it adds an extra node that the other IPI platforms do not need. Second, this node is a single point of failure. The Baremetal platform has faced the same issues and they have solved them with a few virtual IP addresses managed by keepalived in combination with coredns static pod running on every node using the mDNS protocol to update records as new nodes are added or removed and a similar static pod haproxy to load balance the control plane internally. The VIPs are defined here in the installer and they use the PlatformStatus field to be passed to the necessary machine-config-operator fields: openshift/api#374 The Bare Metal IPI Networking Infrastructure document is applicable here as well: https://github.com/openshift/installer/blob/master/docs/design/baremetal/networking-infrastructure.md There is also a great opportunity to share some of the configuration files and scripts here. This change needs several other pull requests: Keepalived plus the coredns & haproxy static pods in the MCO: openshift/machine-config-operator#740 Passing the API and DNS VIPs through the installer: openshift#1998 Co-authored-by: Emilio Garcia <[email protected]> Co-authored-by: John Trowbridge <[email protected]> Co-authored-by: Martin Andre <[email protected]> Co-authored-by: Tomas Sedovic <[email protected]> Massive thanks to the Bare Metal and oVirt people!

The experimental OpenStack backend used to create an extra server running DNS and load balancer services that the cluster needed. OpenStack does not always come with DNSaaS or LBaaS so we had to provide the functionality the OpenShift cluster depends on (e.g. the etcd SRV records, the api-int records & load balancing, etc.). This approach is undesirable for two reasons: first, it adds an extra node that the other IPI platforms do not need. Second, this node is a single point of failure. The Baremetal platform has faced the same issues and they have solved them with a few virtual IP addresses managed by keepalived in combination with coredns static pod running on every node using the mDNS protocol to update records as new nodes are added or removed and a similar static pod haproxy to load balance the control plane internally. The VIPs are defined here in the installer and they use the PlatformStatus field to be passed to the necessary machine-config-operator fields: openshift/api#374 The Bare Metal IPI Networking Infrastructure document is applicable here as well: https://github.com/openshift/installer/blob/master/docs/design/baremetal/networking-infrastructure.md There is also a great opportunity to share some of the configuration files and scripts here. This change needs several other pull requests: Keepalived plus the coredns & haproxy static pods in the MCO: openshift/machine-config-operator#740 Co-authored-by: Emilio Garcia <[email protected]> Co-authored-by: John Trowbridge <[email protected]> Co-authored-by: Martin Andre <[email protected]> Co-authored-by: Tomas Sedovic <[email protected]> Massive thanks to the Bare Metal and oVirt people!

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 13, 2019

openshift-ci-robot requested review from LorbusChris and ashcrow May 13, 2019 19:02

trown mentioned this pull request May 13, 2019

openstack: remove SRV records from service VM openshift/installer#1746

Closed

trown force-pushed the openstack-mdns branch from 1a6110f to 6d6319f Compare May 14, 2019 12:38

cgwalters added the 4.2 label May 14, 2019

cgwalters reviewed May 14, 2019

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 14, 2019

trown force-pushed the openstack-mdns branch from 6d6319f to d942094 Compare May 15, 2019 15:48

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 15, 2019

trown force-pushed the openstack-mdns branch from d942094 to b7f9ad4 Compare May 15, 2019 17:00

trown mentioned this pull request May 15, 2019

openstack: Add mdns plugin for coredns static pods openshift/coredns#12

Closed

abhinavdahiya reviewed May 16, 2019

View reviewed changes

templates/master/00-master/openstack/files/openstack-coredns.yaml Outdated

Copy link

Contributor

abhinavdahiya May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove kni

abhinavdahiya reviewed May 16, 2019

View reviewed changes

templates/master/00-master/openstack/files/openstack-coredns.yaml Outdated

Copy link

Contributor

abhinavdahiya May 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These requests are pretty high.

bcrochet mentioned this pull request May 29, 2019

baremetal: Add coredns, mdns-publisher, haproxy, keepalived static pods #795

Merged

tomassedovic mentioned this pull request Jul 1, 2019

data/data/openstack: Open UDP 5353 (mdns) port openshift/installer#1925

Merged

tomassedovic mentioned this pull request Jul 30, 2019

openstack: Add OpenStack to the platform selection openshift/installer#2036

Merged

openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 30, 2019

mandre force-pushed the openstack-mdns branch from 9fe3f47 to ff19c99 Compare July 30, 2019 18:35

John Trowbridge and others added 5 commits July 30, 2019 23:30

openstack: Run Coredns, Haproxy, Mdns, Keepalived, and DHCP in static…

9fe8641

… pods In order to have a more fault tolerant networking architecture, we are replacing the functionality of the service vm with a number of static pod resources that are run on master and worker nodes.

openstack: Configure Templates to use Baremetal-RuntimeCFG

af2d105

We use Baremetal-RuntimeCfg to clean up our code, as well as align our architecture much more closely with that of the Baremetal Team.

openstack: Fix the tests

0d6e9f2

mandre force-pushed the openstack-mdns branch from ff19c99 to 759702e Compare July 30, 2019 21:35

openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 30, 2019

cgwalters reviewed Aug 1, 2019

View reviewed changes

openshift-ci-robot assigned cgwalters Aug 1, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 1, 2019

openshift-merge-robot merged commit 7f52bd8 into openshift:master Aug 2, 2019

Fedosin mentioned this pull request Aug 7, 2019

Revert "openstack: Allow to run MCO changes independently of the installer patches" #1040

Merged

openstack: Add coredns, mdns-publisher and haproxy static pods #740

openstack: Add coredns, mdns-publisher and haproxy static pods #740

Uh oh!

Conversation

trown commented May 13, 2019

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

trown commented May 15, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mandre commented Jul 31, 2019

Uh oh!

mandre commented Jul 31, 2019

Uh oh!

tomassedovic commented Jul 31, 2019

Uh oh!

tomassedovic commented Jul 31, 2019

Uh oh!

tomassedovic commented Jul 31, 2019

Uh oh!

tomassedovic commented Aug 1, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Aug 1, 2019

Uh oh!

openshift-ci-robot commented Aug 1, 2019

Uh oh!

cgwalters commented Aug 1, 2019

Uh oh!

openshift-bot commented Aug 1, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants