Skip to content

[release-4.16] OCPBUGS-45931: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup #2158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
openshift-cherrypick-robot wants to merge 4 commits into from
Closed

[release-4.16] OCPBUGS-45931: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup #2158

openshift-cherrypick-robot wants to merge 4 commits into from

Conversation

@openshift-cherrypick-robot
Copy link

@openshift-cherrypick-robot openshift-cherrypick-robot commented Dec 9, 2024

This is an automated cherry-pick of #2126

/assign haircommander

none

haircommander and others added 4 commits December 9, 2024 21:54
@haircommander
UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from mi…
1cea012 
…ssing cpuset cgroup

on None cpumanager policy, cgroupv2, and systemd cgroup manager, kubelet
could get into a situation where it believes the cpuset cgroup was created
(by libcontainer in the cgroupfs) but systemd has deleted it, as it wasn't requested
to create it. This causes one unnecessary restart, as kubelet fails with

`failed to initialize top level QOS containers: root container [kubepods] doesn't exist.`

This only causes one restart because the kubelet skips recreating the cgroup
if it already exists, but it's still a bother and is fixed this way

adapted version of kubernetes@c51195d, avoid merge conflict
this can be dropped in the 4.19 rebase

Signed-off-by: Peter Hunt <[email protected]>
@haircommander
UPSTREAM: <carry>: kubelet/cm: move CPU reading from cm to cm/cpumanager
c3b5deb 
Adapted from kubernetes@77d03e4 to avoid merge conflict
This can be dropped in the 4.19 rebase

Authored-by: Francesco Romani <[email protected]>
Signed-off-by: Peter Hunt <[email protected]>
@ffromani
UPSTREAM: 125923: e2e_node: add a test to verify the kubelet starts
177868e 
with systemd cgroup driver and cpumanager none policy.

This was originally planned to be a correctness check for
https://issues.k8s.io/125923, but it was difficult to reproduce the bug,
so it's now a regression test against it.

Signed-off-by: Francesco Romani <[email protected]>
Signed-off-by: Peter Hunt <[email protected]>
@haircommander
UPSTREAM: 125923: e2e_node: use restart instead of start stop
2db5e12 
Signed-off-by: Peter Hunt <[email protected]>
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Dec 9, 2024
Merged
@openshift-ci-robot openshift-ci-robot added the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Dec 9, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 9, 2024

@openshift-cherrypick-robot: Jira Issue OCPBUGS-43820 has been cloned as Jira Issue OCPBUGS-45931. Will retitle bug to link to clone.
/retitle [release-4.16] OCPBUGS-45931: UPSTREAM: : kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup

Details

In response to this:

This is an automated cherry-pick of #2126

/assign haircommander

none

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 9, 2024

@openshift-cherrypick-robot: the contents of this pull request could not be automatically validated.

The following commits are valid:

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

@openshift-ci openshift-ci bot changed the title [release-4.16] OCPBUGS-43820: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup [release-4.16] OCPBUGS-45931: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup Dec 9, 2024
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Dec 9, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 9, 2024

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-45931, which is invalid:

  • expected dependent Jira Issue OCPBUGS-43820 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #2126

/assign haircommander

none

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from deads2k and rphillips December 9, 2024 21:55
@openshift-ci
Copy link

openshift-ci bot commented Dec 9, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: openshift-cherrypick-robot
Once this PR has been reviewed and has the lgtm label, please assign bertinatto for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@haircommander
Copy link
Member

haircommander commented Jan 9, 2025

/retest

@sreber84
Copy link

sreber84 commented Jan 16, 2025

/test unit

@openshift-ci
Copy link

openshift-ci bot commented Jan 16, 2025

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-agnostic-ovn-cmd 2db5e12 link false /test e2e-agnostic-ovn-cmd
ci/prow/verify 2db5e12 link true /test verify
ci/prow/unit 2db5e12 link true /test unit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@haircommander
Copy link
Member

haircommander commented Feb 6, 2025

/close

in favor of #2200

@openshift-ci openshift-ci bot closed this Feb 6, 2025
@openshift-ci
Copy link

openshift-ci bot commented Feb 6, 2025

@haircommander: Closed this PR.

Details

In response to this:

/close

in favor of #2200

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Feb 6, 2025

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-45931. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #2126

/assign haircommander

none

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-bot bot added a commit that referenced this pull request Feb 7, 2025
@openshift-merge-bot
Merge pull request #2200 from haircommander/cherry-pick-2126-to-relea…
148a389 
…se-4.16

 [release-4.16] OCPBUGS-45931: UPSTREAM: <carry>: kubelet/cm: fix bug where kubelet restarts from missing cpuset cgroup #2158
@openshift-ci-robot openshift-ci-robot mentioned this pull request Feb 28, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@deads2k deads2k Awaiting requested review from deads2k

@rphillips rphillips Awaiting requested review from rphillips

Assignees

@haircommander haircommander

Labels

backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@openshift-cherrypick-robot @openshift-ci-robot @haircommander @sreber84 @ffromani