Skip to content

OCPBUGS-60546: UPSTREAM: 135680: test/e2e: increase memory limits in pod resize tests #2533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dgrisonnet wants to merge 1 commit into
base: master
Choose a base branch
from
Open

OCPBUGS-60546: UPSTREAM: 135680: test/e2e: increase memory limits in pod resize tests #2533

dgrisonnet wants to merge 1 commit into from
+6 −6

Conversation

@dgrisonnet
Copy link
Member

@dgrisonnet dgrisonnet commented Dec 9, 2025

Pick kubernetes#135680 to fix downstream flakes with pod inplace resize tests.

@dgrisonnet
UPSTREAM: 135680: test/e2e: increase memory limits in pod resize tests
bd75dce 
The pod resize e2e tests use memory limits as low as 20Mi for Guaranteed
QoS pods. On OpenShift/CRI-O, the container runtime (runc) runs inside
the pod's cgroup and requires ~20-22MB of memory during container
creation and restart operations. This causes intermittent OOM kills
when the pod's memory limit is at or below runc's memory footprint.

This issue does not occur on containerd-based clusters because
containerd's shim runs outside the pod's cgroup by default (ShimCgroup=""),
so runc's memory is not charged against the pod's limit.

Increase memory limits to provide sufficient headroom for runc:
- originalMem: 20Mi -> 35Mi
- reducedMem: 15Mi -> 30Mi
- increasedMem: 25Mi -> 40Mi

The test validates resize behavior, not minimal memory limits, so
larger values do not reduce test coverage.

Signed-off-by: Damien Grisonnet <[email protected]>
@openshift-ci-robot openshift-ci-robot added backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Dec 9, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 9, 2025

@dgrisonnet: This pull request references Jira Issue OCPBUGS-60546, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @asahay19

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Pick kubernetes#135680 to fix downstream flakes with pod inplace resize tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Dec 9, 2025

@dgrisonnet: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

@openshift-ci
Copy link

openshift-ci bot commented Dec 9, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dgrisonnet
Once this PR has been reviewed and has the lgtm label, please assign p0lyn0mial for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@bitoku
Copy link

bitoku commented Dec 9, 2025

/payload-aggregate periodic-ci-openshift-release-master-nightly-4.21-e2e-gcp-ovn-runc 10

@openshift-ci
Copy link

openshift-ci bot commented Dec 9, 2025

@bitoku: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.21-e2e-gcp-ovn-runc

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/cb649f00-d504-11f0-8baf-64f82bef5ef1-0

@openshift-ci
Copy link

openshift-ci bot commented Dec 9, 2025

@dgrisonnet: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-hypershift bd75dce link true /test e2e-aws-ovn-hypershift

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@rphillips
Copy link

rphillips commented Dec 9, 2025

The job did flake once in the aggregate.

@dgrisonnet
Copy link
Member Author

dgrisonnet commented Dec 10, 2025

/payload-aggregate periodic-ci-openshift-release-master-nightly-4.21-e2e-gcp-ovn-runc 10

@openshift-ci
Copy link

openshift-ci bot commented Dec 10, 2025

@dgrisonnet: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.21-e2e-gcp-ovn-runc

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ef076e00-d5af-11f0-8c43-a70d918b954d-0

@bitoku
Copy link

bitoku commented Dec 10, 2025

The job did flake once in the aggregate.

Though it failed, this doesn't look OOM.
We may need to do something else, but I believe this PR still be legit.

@haircommander
Copy link
Member

haircommander commented Dec 10, 2025

/retest

yeah failed to verify cpu limit cgroup value: value of cgroup "/sys/fs/cgroup/cpu.max" for container "c1-init" was ""; expected one of ["1500 100000" "2000 100000"] looks like a different race that i've not seen yet

@haircommander
Copy link
Member

haircommander commented Dec 10, 2025

/payload-aggregate periodic-ci-openshift-release-master-nightly-4.21-e2e-gcp-ovn-runc 20

@openshift-ci
Copy link

openshift-ci bot commented Dec 10, 2025

@haircommander: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.21-e2e-gcp-ovn-runc

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f2ee8f70-d600-11f0-8127-0592e288e31d-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@asahay19 asahay19 Awaiting requested review from asahay19

@bertinatto bertinatto Awaiting requested review from bertinatto

@jerpeter1 jerpeter1 Awaiting requested review from jerpeter1

Assignees

No one assigned

Labels

backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dgrisonnet @openshift-ci-robot @bitoku @rphillips @haircommander