upi/metal various fixes #2695

sdodson · 2019-11-20T18:37:08Z

While attempting to reduce the number of times we fail to install during the e2e-metal job I've encounter a few different issues that contribute to the job's instability.

Only enable the first interface, the network topology that we've selected provides an LACP bond across available interfaces, however if the host hasn't been configured to use that properly it will attempt to bring up each interface using DHCP and that adds substantial delay to the boot process potentially leading to failure. I've removed this fix for now as I'm attempting to use a different instance type which may have different interface names than those on c1.small.x86
Specify the facility for all instances, without this I was seeing masters across SJC1 and EWR1 datacenters which is not compatible with etcd latency requirements

sdodson · 2019-11-20T18:37:37Z

/test e2e-metal

sdodson · 2019-11-20T22:22:25Z

/test e2e-metal

sdodson · 2019-11-21T14:30:27Z

/test e2e-metal

sdodson · 2019-11-21T15:27:53Z

/test e2e-metal

sdodson · 2019-11-21T19:37:48Z

/test e2e-metal

sdodson · 2019-11-21T20:37:30Z

/test e2e-metal

sdodson · 2019-11-22T13:12:20Z

/test e2e-metal

sdodson · 2019-11-22T14:35:42Z

/test e2e-metal

sdodson · 2019-11-22T20:33:00Z

/test e2e-metal

sdodson · 2019-11-22T21:31:14Z

/test e2e-metal

sdodson · 2019-11-23T00:26:17Z

/test e2e-metal

sdodson · 2019-11-24T16:05:21Z

/retitle WIP Bug 1773108: upi/metal various fixes

openshift-ci-robot · 2019-11-24T16:05:24Z

@sdodson: This pull request references Bugzilla bug 1773108, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "4.3.0" instead
expected dependent Bugzilla bug 1775388 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ASSIGNED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

WIP Bug 1773108: upi/metal various fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-11-24T16:12:50Z

@sdodson: This pull request references Bugzilla bug 1773108, which is invalid:

expected dependent Bugzilla bug 1775388 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ASSIGNED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

WIP Bug 1773108: upi/metal various fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sdodson · 2019-11-24T16:13:39Z

/test e2e-metal

sdodson · 2019-11-24T18:52:19Z

/bugzilla refresh

openshift-ci-robot · 2019-11-24T18:52:21Z

@sdodson: This pull request references Bugzilla bug 1773108, which is invalid:

expected dependent Bugzilla bug 1775388 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ASSIGNED instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-12-12T19:13:23Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign sdodson
You can assign the PR to them by writing /assign @sdodson in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2019-12-12T19:14:08Z

@sdodson: This pull request references Bugzilla bug 1773108, which is valid.

Details

In response to this:

WIP Bug 1773108: upi/metal various fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sdodson · 2019-12-12T19:17:15Z

/test e2e-metal

I was seeing masters within a cluster being splayed across SJC1 and EWR1 datacenters and that can contribute to etcd latency which may lead to some of the E2E failures we're seeing. Future improvement could be to provision the bootstrap host, determine its facility and then ensure all the remaining hosts are provisioned into the same facility.

I was observing up to several minutes of delay in bringing up the network during coreos installation phase. This cuts down on that time by a bit but doesn't seem to completely prevent NetworkManager-wait-online.service timeouts as it doesn't seem to carry through to the installed OS.

openshift-ci-robot · 2019-12-20T13:21:09Z

@sdodson: This pull request references Bugzilla bug 1773108, which is invalid:

expected the bug to be open, but it isn't
expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is CLOSED (DUPLICATE) instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

WIP Bug 1773108: upi/metal various fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-12-20T13:21:26Z

@sdodson: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

upi/metal various fixes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sdodson · 2019-12-20T13:21:41Z

/test e2e-metal

sdodson · 2019-12-20T14:18:10Z

/test e2e-metal

sdodson · 2019-12-23T15:06:26Z

/test e2e-metal

sdodson · 2020-01-06T21:02:54Z

/test e2e-metal

sdodson · 2020-01-31T19:48:08Z

/test e2e-metal

sdodson · 2020-01-31T19:53:24Z

error: invalid configuration: tests[15]: non-literal test found in fully-resolved configuration
/shrug

/test e2e-metal

abhinavdahiya · 2020-02-10T18:41:08Z

/retest

openshift-ci-robot · 2020-04-22T00:52:39Z

@sdodson: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-ovirt	`4c6df03`	link	`/test e2e-ovirt`
ci/prow/e2e-libvirt	`4c6df03`	link	`/test e2e-libvirt`
ci/prow/e2e-metal	`4c6df03`	link	`/test e2e-metal`
ci/prow/e2e-aws-scaleup-rhel7	`4c6df03`	link	`/test e2e-aws-scaleup-rhel7`
ci/prow/tf-lint	`4c6df03`	link	`/test tf-lint`
ci/prow/yaml-lint	`4c6df03`	link	`/test yaml-lint`
ci/prow/shellcheck	`4c6df03`	link	`/test shellcheck`
ci/prow/e2e-aws-upgrade	`4c6df03`	link	`/test e2e-aws-upgrade`
ci/prow/unit	`4c6df03`	link	`/test unit`
ci/prow/golint	`4c6df03`	link	`/test golint`
ci/prow/gofmt	`4c6df03`	link	`/test gofmt`
ci/prow/images	`4c6df03`	link	`/test images`
ci/prow/govet	`4c6df03`	link	`/test govet`
ci/prow/verify-vendor	`4c6df03`	link	`/test verify-vendor`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

abhinavdahiya · 2020-05-08T23:04:08Z

/close

closing due to inactivity

openshift-ci-robot · 2020-05-08T23:04:22Z

@abhinavdahiya: Closed this PR.

Details

In response to this:

/close

closing due to inactivity

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 20, 2019

openshift-ci-robot requested review from jhixson74 and mtnbikenc November 20, 2019 18:38

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 20, 2019

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 23, 2019

sdodson force-pushed the metal-facilities branch from 1a6d1c7 to dc5aa46 Compare November 24, 2019 15:57

sdodson changed the title ~~WIP upi/metal: Set the facility~~ WIP upi/metal: Various fixes Nov 24, 2019

openshift-ci-robot changed the title ~~WIP upi/metal: Various fixes~~ WIP Bug 1773108: upi/metal various fixes Nov 24, 2019

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Nov 24, 2019

sdodson mentioned this pull request Nov 24, 2019

WIP Bug 1773108: upi/metal Only configure the first interface #2688

Closed

sdodson force-pushed the metal-facilities branch from dc5aa46 to 61bcd58 Compare November 24, 2019 16:13

openshift-ci-robot removed the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 12, 2019

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Dec 13, 2019

sdodson added 2 commits December 20, 2019 08:11

sdodson force-pushed the metal-facilities branch from 85fa20b to 4c6df03 Compare December 20, 2019 13:14

sdodson changed the title ~~WIP Bug 1773108: upi/metal various fixes~~ upi/metal various fixes Dec 20, 2019

openshift-ci-robot removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Dec 20, 2019

openshift-ci-robot added the ¯\_(ツ)_/¯ ¯\\\_(ツ)_/¯ label Jan 31, 2020

openshift-ci-robot closed this May 8, 2020

upi/metal various fixes #2695

upi/metal various fixes #2695

Uh oh!

Conversation

sdodson commented Nov 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sdodson commented Nov 20, 2019

Uh oh!

sdodson commented Nov 20, 2019

Uh oh!

sdodson commented Nov 21, 2019

Uh oh!

sdodson commented Nov 21, 2019

Uh oh!

sdodson commented Nov 21, 2019

Uh oh!

sdodson commented Nov 21, 2019

Uh oh!

sdodson commented Nov 22, 2019

Uh oh!

sdodson commented Nov 22, 2019

Uh oh!

sdodson commented Nov 22, 2019

Uh oh!

sdodson commented Nov 22, 2019

Uh oh!

sdodson commented Nov 23, 2019

Uh oh!

sdodson commented Nov 24, 2019

Uh oh!

openshift-ci-robot commented Nov 24, 2019

Uh oh!

openshift-ci-robot commented Nov 24, 2019

Uh oh!

sdodson commented Nov 24, 2019

Uh oh!

sdodson commented Nov 24, 2019

Uh oh!

openshift-ci-robot commented Nov 24, 2019

Uh oh!

openshift-ci-robot commented Dec 12, 2019

Uh oh!

openshift-ci-robot commented Dec 12, 2019

Uh oh!

sdodson commented Dec 12, 2019

Uh oh!

openshift-ci-robot commented Dec 20, 2019

Uh oh!

openshift-ci-robot commented Dec 20, 2019

Uh oh!

sdodson commented Dec 20, 2019

Uh oh!

sdodson commented Dec 20, 2019

Uh oh!

sdodson commented Dec 23, 2019

Uh oh!

sdodson commented Jan 6, 2020

Uh oh!

sdodson commented Jan 31, 2020

Uh oh!

sdodson commented Jan 31, 2020

Uh oh!

abhinavdahiya commented Feb 10, 2020

Uh oh!

openshift-ci-robot commented Apr 22, 2020

Uh oh!

abhinavdahiya commented May 8, 2020

Uh oh!

openshift-ci-robot commented May 8, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sdodson commented Nov 20, 2019 •

edited

Loading