Bug 1825976: cherrypick etcd-quorum-guard refactoring to 4.4 #1655

vrutkovs · 2020-04-15T12:16:26Z

- What I did
Cherrypicked #1552 and #1648 to release-4.4

- How to verify it
Run install / upgrade. etcd-quorum-guard pods should not fail or leak memory

- Description for the changelog
Cherrypicked etcd-quorum-guard refactoring on 4.4

Prepare the script once so that readinessProbe command would not look for certificate path every time. This also uses downward API to avoid looking for certificates and fetch them from defined locations

Avoid setting hostNetwork for etcd-quorum-guard

Ensure `curl` command has NSS_SDB_USE_CACHE env var set

openshift-ci-robot · 2020-04-15T12:16:29Z

@vrutkovs: This pull request references Bugzilla bug 1824137, which is invalid:

expected dependent Bugzilla bug 1823677 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1824137: cherrypick etcd-quorum-guard refactoring to 4.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yuqi-zhang · 2020-04-15T16:02:16Z

/retest
/approve

openshift-ci-robot · 2020-04-15T16:02:53Z

@yuqi-zhang: The /retest command does not accept any targets.
The following commands are available to trigger jobs:

/test e2e-aws
/test e2e-aws-disruptive
/test e2e-aws-scaleup-rhel7
/test e2e-gcp-op
/test e2e-gcp-upgrade
/test e2e-metal-ipi
/test e2e-openstack
/test e2e-ovirt
/test e2e-vsphere
/test images
/test unit
/test verify

Use /test all to run all jobs.

Details

In response to this:

/retest
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2020-04-16T11:56:58Z

/retest

kikisdeliveryservice · 2020-04-16T17:40:21Z

/bugzilla refresh

openshift-ci-robot · 2020-04-16T17:40:24Z

@kikisdeliveryservice: This pull request references Bugzilla bug 1824137, which is invalid:

expected dependent Bugzilla bug 1823677 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice · 2020-04-16T18:40:14Z

/retest

kikisdeliveryservice · 2020-04-16T18:40:38Z

/assign @hexfusion

runcom · 2020-04-17T13:15:58Z

/retest
/lgtm

openshift-ci-robot · 2020-04-17T13:18:26Z

@runcom: The /retest command does not accept any targets.
The following commands are available to trigger jobs:

/test e2e-aws
/test e2e-aws-disruptive
/test e2e-aws-scaleup-rhel7
/test e2e-gcp-op
/test e2e-gcp-upgrade
/test e2e-metal-ipi
/test e2e-openstack
/test e2e-ovirt
/test e2e-vsphere
/test images
/test unit
/test verify

Use /test all to run all jobs.

Details

In response to this:

/retest
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-04-17T13:18:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: runcom, vrutkovs, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [runcom,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-04-17T13:39:48Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-17T19:43:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-17T20:09:28Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-17T22:19:29Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-18T00:16:50Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-04-18T01:01:31Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci-robot · 2020-04-18T01:02:13Z

@openshift-bot: This pull request references Bugzilla bug 1824137, which is invalid:

expected dependent Bugzilla bug 1823677 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

eparis · 2020-04-20T16:21:55Z

/hold

eparis · 2020-04-20T16:24:42Z

This PR conflates 2 distinct issue.

a cosmetic issue about traffic counts that, as best I can tell, hurts nothing
an oddity in NSS/kernel negative dentries which does have a negative impact.

Can we at least get this PR and bug to talk (primarily) about #2 instead of #1?

Let's not combine unrelated things in backports in the future, please. i know it's more 'paperwork' but please don't.

Also, how does moving from hostNetwork to SDN affect things if the SDN is down? previously I didn't need SDN to get quorum guard running. Now I will. What affects is that going to have? How do we test is this change is detrimental?

After that you may remove the hold.

Please also ping Bob Relyea and Nikos about the upstream nss/kernel issue to get it back on their radar.

vrutkovs · 2020-04-20T16:46:55Z

a cosmetic issue about traffic counts that, as best I can tell, hurts nothing

Correct

an oddity in NSS/kernel negative dentries which does have a negative impact.

There is no negative impact of this in 4.4 (there has been in 4.5). export NSS... needs to be moved in the script which is being run during health checks.

Also, how does moving from hostNetwork to SDN affect things if the SDN is down?

etcd-quorum-guard would be restarted if SDN on master is down. If its down then API container is not reachable anyway

What affects is that going to have? How do we test is this change is detrimental?

I tested it with running minor and major upgrade - SDN gets disrupted and etcd-quorum-guard survives that.

NSS env var is orthogonal to this PR - it merely moves it in the script which runs curl as this part is being refactored

hexfusion · 2020-04-20T17:36:12Z

Given the above, while I see this as a possible improvement in networking metrics moving quorum-guard to SDN. But I feel like we are missing coverage on full understanding on corner cases. With the CI data that exists, we should be able to better understand this.

So let's spend more time in that validation and this can drop into z-stream if we see the net gain.

ashcrow · 2020-04-22T13:39:22Z

/retest

sdodson · 2020-04-24T13:11:13Z

/bugzilla refresh
(to pick up retargetting to 4.4.z)

openshift-ci-robot · 2020-04-24T13:11:18Z

@sdodson: This pull request references Bugzilla bug 1825976, which is invalid:

expected the bug to target the "4.4.0" release, but it targets "4.4.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh
(to pick up retargetting to 4.4.z)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

runcom · 2020-05-07T19:54:03Z

/bugzilla refresh

openshift-ci-robot · 2020-05-07T19:54:07Z

@runcom: This pull request references Bugzilla bug 1825976, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.4.z) matches configured target release for branch (4.4.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1825967 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA))
dependent Bugzilla bug 1825967 targets the "4.5.0" release, which is one of the valid target releases: 4.5.0, 4.5.z
bug has dependents

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

runcom · 2020-05-07T19:54:41Z

@hexfusion bz is now valid, pending your hold and approval to get in for the z stream

LorbusChris · 2020-05-09T11:05:39Z

/cherry-pick fcos

openshift-cherrypick-robot · 2020-05-09T11:05:40Z

@LorbusChris: once the present PR merges, I will cherry-pick it on top of fcos in a new PR and assign it to you.

Details

In response to this:

/cherry-pick fcos

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

LorbusChris · 2020-05-09T14:54:56Z

/retest

vrutkovs · 2020-05-11T09:57:28Z

/retest

ashcrow · 2020-05-11T12:50:01Z

/retest

sinnykumari · 2020-05-12T06:23:59Z

/retest

openshift-ci-robot · 2020-05-12T07:42:32Z

@vrutkovs: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-gcp-upgrade	`6836bb6`	link	`/test e2e-gcp-upgrade`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sdodson · 2020-05-13T19:12:09Z

My understanding is that our intent is to close this and revert the upstream change. As such not giving cherry-pick-approved despite this bug being marked high severity.

hexfusion · 2020-05-13T19:51:47Z

/close

After conversations with @eparis this is not what we want. Essentially this would run quorum-guard checks through SDN.

openshift-ci-robot · 2020-05-13T19:52:00Z

@hexfusion: Closed this PR.

Details

In response to this:

/close

After conversations with @eparis this is not what we want. Essentially this would run quorum-guard checks through SDN.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Vadim Rutkovsky added 3 commits April 15, 2020 14:10

etcd-quorum-guard: prepare readniess script in the pod

3944a69

Prepare the script once so that readinessProbe command would not look for certificate path every time. This also uses downward API to avoid looking for certificates and fetch them from defined locations

etcd quorum guard: don't set hostNetwork

95f7408

Avoid setting hostNetwork for etcd-quorum-guard

etcdquorumguard_deployment: pass NSS_SDB_USE_CACHE=no to curl

6836bb6

Ensure `curl` command has NSS_SDB_USE_CACHE env var set

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Apr 15, 2020

openshift-ci-robot requested review from cgwalters and yuqi-zhang April 15, 2020 12:16

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 15, 2020

kikisdeliveryservice changed the title ~~Bug 1824137: cherrypick etcd-quorum-guard refactoring to 4.4~~ [release-4.4] Bug 1824137: cherrypick etcd-quorum-guard refactoring to 4.4 Apr 16, 2020

openshift-ci-robot assigned hexfusion Apr 16, 2020

openshift-ci-robot assigned runcom Apr 17, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 17, 2020

openshift-ci-robot added the bugzilla/high label Apr 18, 2020

mfojtik added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Apr 20, 2020

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 20, 2020

eparis removed the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Apr 20, 2020

openshift-ci-robot added bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. and removed bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Apr 24, 2020

openshift-ci-robot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. and removed bugzilla/high labels Apr 29, 2020

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label May 7, 2020

openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label May 7, 2020

openshift-ci-robot closed this May 13, 2020

Bug 1825976: cherrypick etcd-quorum-guard refactoring to 4.4 #1655

Bug 1825976: cherrypick etcd-quorum-guard refactoring to 4.4 #1655

Uh oh!

Conversation

vrutkovs commented Apr 15, 2020

Uh oh!

openshift-ci-robot commented Apr 15, 2020

Uh oh!

yuqi-zhang commented Apr 15, 2020

Uh oh!

openshift-ci-robot commented Apr 15, 2020

Uh oh!

vrutkovs commented Apr 16, 2020

Uh oh!

kikisdeliveryservice commented Apr 16, 2020

Uh oh!

openshift-ci-robot commented Apr 16, 2020

Uh oh!

kikisdeliveryservice commented Apr 16, 2020

Uh oh!

kikisdeliveryservice commented Apr 16, 2020

Uh oh!

runcom commented Apr 17, 2020

Uh oh!

openshift-ci-robot commented Apr 17, 2020

Uh oh!

openshift-ci-robot commented Apr 17, 2020

Uh oh!

openshift-bot commented Apr 17, 2020

Uh oh!

openshift-bot commented Apr 17, 2020

Uh oh!

openshift-bot commented Apr 17, 2020

Uh oh!

openshift-bot commented Apr 17, 2020

Uh oh!

openshift-bot commented Apr 18, 2020

Uh oh!

openshift-bot commented Apr 18, 2020

Uh oh!

openshift-ci-robot commented Apr 18, 2020

Uh oh!

eparis commented Apr 20, 2020

Uh oh!

eparis commented Apr 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vrutkovs commented Apr 20, 2020

Uh oh!

hexfusion commented Apr 20, 2020

Uh oh!

ashcrow commented Apr 22, 2020

Uh oh!

sdodson commented Apr 24, 2020

Uh oh!

openshift-ci-robot commented Apr 24, 2020

Uh oh!

runcom commented May 7, 2020

Uh oh!

openshift-ci-robot commented May 7, 2020

Uh oh!

runcom commented May 7, 2020

Uh oh!

LorbusChris commented May 9, 2020

Uh oh!

openshift-cherrypick-robot commented May 9, 2020

Uh oh!

LorbusChris commented May 9, 2020

Uh oh!

vrutkovs commented May 11, 2020

Uh oh!

ashcrow commented May 11, 2020

Uh oh!

sinnykumari commented May 12, 2020

Uh oh!

openshift-ci-robot commented May 12, 2020

Uh oh!

sdodson commented May 13, 2020

Uh oh!

hexfusion commented May 13, 2020

eparis commented Apr 20, 2020 •

edited

Loading