Bug 2071139: delete default ingress pod if it is scheduled where another router pod already is by deads2k · Pull Request #720 · openshift/cluster-ingress-operator

deads2k · 2022-03-14T18:13:26Z

Anti-affinity in the scheduler has a bug and isn't properly spreading pods. This fits a new library-go controller into the ingress-operator to delete a mal-scheduled pod. If it finds two non-deleted pods using the selector on the same node, then the oldest pod is deleted. Oldest is chosen so that a deployment controller could still make progress. This different that is commonly done to preserve the oldest to disrupt the fewest connections, but it's better to avoid blocking the deployment controller.

I chose to start with ingress because it has the biggest multiplier and this doesn't fully resolve the bug for this component, it only impacts the default ingress.

/hold
This PR is proof for openshift/library-go#1330

deads2k · 2022-03-14T18:13:52Z

/hold

deads2k · 2022-03-24T18:06:58Z

/test all

deads2k · 2022-03-31T12:11:22Z

/retest

deads2k · 2022-03-31T13:34:22Z

/hold cancel

The controller and unit tests merged to library-go, so this bump is now real.

deads2k · 2022-03-31T17:18:05Z

/retest

deads2k · 2022-03-31T18:01:06Z

    "Operator \"cluster-monitoring-operator\" produces more watch requests than expected: watchrequestcount=104, upperbound=96, ratio=1.0833333333333333",

/retest

deads2k · 2022-03-31T19:24:22Z

/retest

deads2k · 2022-04-01T13:11:51Z

/hold

good news: it works. Bad news, unit test https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-ingress-operator/720/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws/1509612592396505088/artifacts/e2e-aws/gather-extra/artifacts/pods/openshift-ingress-operator_ingress-operator-697bc657df-wlc2g_ingress-operator_previous.log

deads2k · 2022-04-01T13:18:41Z

Fix here openshift/library-go#1344, will pick when it lands.

…d already is

deads2k · 2022-04-01T14:58:31Z

bumped

/hold cancel

Miciah · 2022-04-01T19:58:09Z

Thanks!
/lgtm

openshift-ci · 2022-04-01T19:59:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Miciah]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2022-04-01T20:15:04Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-01T20:41:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-04-01T21:49:43Z

@deads2k: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Miciah · 2022-04-01T23:43:35Z

/retitle Bug 2071139: delete default ingress pod if it is scheduled where another router pod already is

openshift-ci · 2022-04-01T23:43:43Z

@deads2k: This pull request references Bugzilla bug 2071139, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.11.0) matches configured target release for branch (4.11.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @lihongan

Details

In response to this:

Bug 2071139: delete default ingress pod if it is scheduled where another router pod already is

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-04-01T23:46:13Z

@deads2k: All pull requests linked via external trackers have merged:

openshift/cluster-ingress-operator#720

Bugzilla bug 2071139 has been moved to the MODIFIED state.

Details

In response to this:

Bug 2071139: delete default ingress pod if it is scheduled where another router pod already is

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 14, 2022

openshift-ci bot requested review from candita and rfredette March 14, 2022 18:14

deads2k force-pushed the add-pod branch 2 times, most recently from 4e3a09a to 2779ca2 Compare March 16, 2022 19:02

deads2k force-pushed the add-pod branch from f5c6ddc to 7246091 Compare March 24, 2022 16:17

deads2k force-pushed the add-pod branch from 7246091 to 09764e0 Compare March 30, 2022 17:37

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 31, 2022

deads2k force-pushed the add-pod branch from 09764e0 to 5c98540 Compare March 31, 2022 15:31

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 1, 2022

deads2k added 2 commits April 1, 2022 10:57

update vendor

b662651

delete default ingress pod if it is scheduled where another router po…

91dc183

…d already is

deads2k force-pushed the add-pod branch from 5c98540 to 91dc183 Compare April 1, 2022 14:58

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 1, 2022

openshift-ci bot assigned Miciah Apr 1, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 1, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 1, 2022

openshift-ci bot changed the title ~~delete default ingress pod if it is scheduled where another router pod already is~~ Bug 2071139: delete default ingress pod if it is scheduled where another router pod already is Apr 1, 2022

openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Apr 1, 2022

openshift-ci bot requested a review from lihongan April 1, 2022 23:43

openshift-merge-robot merged commit 39b074e into openshift:master Apr 1, 2022

openshift-ci bot mentioned this pull request Apr 8, 2022

Bug 2071139: add pod eviction permission #734

Merged

Conversation

deads2k commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deads2k commented Mar 14, 2022

Uh oh!

deads2k commented Mar 24, 2022

Uh oh!

deads2k commented Mar 31, 2022

Uh oh!

deads2k commented Mar 31, 2022

Uh oh!

deads2k commented Mar 31, 2022

Uh oh!

deads2k commented Mar 31, 2022

Uh oh!

deads2k commented Mar 31, 2022

Uh oh!

deads2k commented Apr 1, 2022

Uh oh!

deads2k commented Apr 1, 2022

Uh oh!

deads2k commented Apr 1, 2022

Uh oh!

Miciah commented Apr 1, 2022

Uh oh!

openshift-ci bot commented Apr 1, 2022

Uh oh!

openshift-bot commented Apr 1, 2022

Uh oh!

openshift-bot commented Apr 1, 2022

Uh oh!

openshift-ci bot commented Apr 1, 2022

Uh oh!

Miciah commented Apr 1, 2022

Uh oh!

openshift-ci bot commented Apr 1, 2022

Uh oh!

openshift-ci bot commented Apr 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

deads2k commented Mar 14, 2022 •

edited

Loading