Bug 2108320: Greatly raise `StartLimitBurst` for `rpm-ostreed.service` #898

cgwalters · 2022-07-18T20:31:01Z

This is a hack to work around the lack of
coreos/rpm-ostree@0556152
for RHEL 8.6 and below.

A recent PR in the MCO openshift/machine-config-operator#3243
tipped things over the edge and we now see failures a lot more often.

cgwalters · 2022-07-18T20:38:50Z

/bugzilla refresh

openshift-ci · 2022-07-18T20:39:09Z

@cgwalters: This pull request references Bugzilla bug 2108320, which is invalid:

expected the bug to target the "4.12.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 2108320: Greatly raise StartLimitBurst for rpm-ostreed.service

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-07-18T20:39:12Z

@cgwalters: This pull request references Bugzilla bug 2108320, which is invalid:

expected the bug to target the "4.12.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2022-07-18T20:39:43Z

/bugzilla refresh

openshift-ci · 2022-07-18T20:39:57Z

@cgwalters: This pull request references Bugzilla bug 2108320, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.12.0) matches configured target release for branch (4.12.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @mike-nguyen

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2022-07-18T20:40:35Z

tests/kola/rpm-ostree/startlimit.sh

+#!/bin/bash
+set -euo pipefail
+# https://github.com/coreos/rpm-ostree/pull/3523/commits/0556152adb14a8e1cdf6c5d6f234aacbe8dd4e3f
+for x in $(seq 10); do rpm-ostree status >/dev/null; done


It's really this easy to reproduce the bug without this patch, just run this shell script one liner.

mike-nguyen · 2022-07-19T12:58:50Z

ci logs show

2022-07-18T21:27:42Z kola: Found non-executable file with shebang: startlimit.sh

This is a hack to work around the lack of coreos/rpm-ostree@0556152 for RHEL 8.6 and below. A recent PR in the MCO openshift/machine-config-operator#3243 tipped things over the edge and we now see failures a lot more often.

cgwalters · 2022-07-19T13:03:09Z

2022-07-18T21:27:42Z kola: Found non-executable file with shebang: startlimit.sh

🙏 thanks! fixed

travier · 2022-07-19T14:00:00Z

LGTM. Pending CI logs to make sure that the test ran.

cgwalters · 2022-07-19T15:55:34Z

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_os/898/pull-ci-openshift-os-master-rhcos-86-build-test-qemu/1549379799590375424/artifacts/test/artifacts/kola/ext.config.rpm-ostree.startlimit/d53520b7-38c3-45a6-aabf-70c01cc187ad/ext.config.rpm-ostree.startlimit.txt

sinnykumari · 2022-07-19T16:50:18Z

periodic-image test failed due to missing base kernel package:

 ERRO[2022-07-19T14:59:31Z] 
  * could not run steps: step rhcos-extensions failed: the build rhcos-extensions failed after 13m16s with reason DockerBuildFailed: Dockerfile build strategy has failed.
[2/4] STEP 8/8: RUN rpm-ostree compose extensions --rootfs...usr/share/rpm-ostree/extensions/ {manifest,extensions}.yaml 
error: parsing extensions.yaml: couldn't find base package kernel

Is this something flake?

cgwalters · 2022-07-19T16:55:10Z

It's an unrelated failure #897 - this PR can't break that

cgwalters · 2022-07-19T16:55:36Z

that prow context is not gating but might as well
/override ci/prow/periodic-images

openshift-ci · 2022-07-19T16:57:16Z

@cgwalters: Overrode contexts on behalf of cgwalters: ci/prow/periodic-images

Details

In response to this:

that prow context is not gating but might as well
/override ci/prow/periodic-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-07-19T16:57:22Z

@cgwalters: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

sinnykumari · 2022-07-19T16:59:36Z

Since, there are no objection so far from reviewers and failing test is unrelated, adding my lgtm to get this in.
/lgtm

openshift-ci · 2022-07-19T17:00:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2022-07-19T17:05:54Z

@cgwalters: All pull requests linked via external trackers have merged:

openshift/os#898

Bugzilla bug 2108320 has been moved to the MODIFIED state.

Details

In response to this:

Bug 2108320: Greatly raise StartLimitBurst for rpm-ostreed.service

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2022-07-19T17:14:34Z

/cherrypick release-4.11

openshift-cherrypick-robot · 2022-07-19T17:15:12Z

@cgwalters: new pull request created: #900

Details

In response to this:

/cherrypick release-4.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from aaradhak and saqibali-2k July 18, 2022 20:32

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 18, 2022

cgwalters changed the title ~~Greatly raise StartLimitBurst for rpm-ostreed.service~~ Bug 2108320: Greatly raise StartLimitBurst for rpm-ostreed.service Jul 18, 2022

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jul 18, 2022

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Jul 18, 2022

openshift-ci bot requested a review from mike-nguyen July 18, 2022 20:40

cgwalters commented Jul 18, 2022

View reviewed changes

Greatly raise StartLimitBurst for rpm-ostreed.service

88d6e1e

This is a hack to work around the lack of coreos/rpm-ostree@0556152 for RHEL 8.6 and below. A recent PR in the MCO openshift/machine-config-operator#3243 tipped things over the edge and we now see failures a lot more often.

cgwalters force-pushed the workaround-rpmostreed branch from 7f8348b to 88d6e1e Compare July 19, 2022 13:02

openshift-ci bot assigned sinnykumari Jul 19, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 19, 2022

openshift-ci bot merged commit f9a029d into openshift:master Jul 19, 2022

openshift-cherrypick-robot mentioned this pull request Jul 19, 2022

[release-4.11] Bug 2108686: Greatly raise StartLimitBurst for rpm-ostreed.service #900

Merged

This was referenced Jul 19, 2022

daemon: Drop workarounds for rpm-ostree bugs openshift/machine-config-operator#3239

Merged

Prep patches for bumping openshift/client-go openshift/machine-config-operator#3251

Merged

sinnykumari mentioned this pull request Jul 21, 2022

daemon: Only remove rollback when going to do an update openshift/machine-config-operator#3253

Closed

Bug 2108320: Greatly raise StartLimitBurst for rpm-ostreed.service #898

Bug 2108320: Greatly raise StartLimitBurst for rpm-ostreed.service #898

Uh oh!

Conversation

cgwalters commented Jul 18, 2022

Uh oh!

cgwalters commented Jul 18, 2022

Uh oh!

openshift-ci bot commented Jul 18, 2022

Uh oh!

openshift-ci bot commented Jul 18, 2022

Uh oh!

cgwalters commented Jul 18, 2022

Uh oh!

openshift-ci bot commented Jul 18, 2022

Uh oh!

cgwalters Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mike-nguyen commented Jul 19, 2022

Uh oh!

cgwalters commented Jul 19, 2022

Uh oh!

travier commented Jul 19, 2022

Uh oh!

cgwalters commented Jul 19, 2022

Uh oh!

sinnykumari commented Jul 19, 2022

Uh oh!

cgwalters commented Jul 19, 2022

Uh oh!

cgwalters commented Jul 19, 2022

Uh oh!

openshift-ci bot commented Jul 19, 2022

Uh oh!

openshift-ci bot commented Jul 19, 2022

Uh oh!

sinnykumari commented Jul 19, 2022

Uh oh!

openshift-ci bot commented Jul 19, 2022

Uh oh!

openshift-ci bot commented Jul 19, 2022

Uh oh!

cgwalters commented Jul 19, 2022

Uh oh!

openshift-cherrypick-robot commented Jul 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Bug 2108320: Greatly raise `StartLimitBurst` for `rpm-ostreed.service` #898

Bug 2108320: Greatly raise `StartLimitBurst` for `rpm-ostreed.service` #898

cgwalters Jul 18, 2022 •

edited

Loading