OTA-1580: Monitortest framework test for oc adm upgrade status #30031
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is to address the failure like [1]. [1]. https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29977/pull-ci-openshift-origin-main-e2e-aws-ovn-microshift-serial/1950320898611351552
|
@hongkailiu: This pull request references OTA-1580 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
openshift-ci
bot
added
the
do-not-merge/work-in-progress
Member
Author
|
/uncc p0lyn0mial sjenning |
openshift-ci
bot
requested review from
petr-muller and
wking
and removed request for
p0lyn0mial and
sjenning
|
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: 9514c30
New tests seen in this PR at sha: 9514c30
|
Member
Author
|
For a failure on aws-ovn-single-node-serial: For a failure on e2e-hypershift-conformance: |
|
Job Failure Risk Analysis for sha: 30dac4a
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: 30dac4a
New tests seen in this PR at sha: 30dac4a
|
Member
Author
|
/retest-required |
Member
Author
|
/retest |
|
Job Failure Risk Analysis for sha: a801047
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: a801047
New tests seen in this PR at sha: a801047
|
wking
reviewed
Aug 2, 2025
| @@ -6,6 +6,7 @@ import ( | |||
| "github.com/openshift/origin/pkg/monitortestframework" | |||
Member
There was a problem hiding this comment.
Unclear if this is related to these changes or not, but the e2e-gcp-ovn-upgrade run's monitor logs contain:
E0731 11:46:20.951174 2139 runtime.go:140] "Observed a panic" panic="May only be called from within a test case" stacktrace=<time="2025-07-31T11:46:20Z" level=info msg=" Starting legacy-kube-apiserver-invariants for kube-apiserver"
goroutine 498 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x9005b58, 0xc00075e000}, {0x73297c0, 0x8f7f930})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:132 +0xbc
time="2025-07-31T11:46:20Z" level=info msg=" Starting legacy-networking-invariants for Networking / cluster-network-operator"
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x9005b58, 0xc00075e000}, {0x73297c0, 0x8f7f930}, {0x0, 0x0, 0x2951560?})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0x116
k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x9005b58, 0xc00075e000}, {0x0, 0x0, 0x0})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
panic({0x73297c0?, 0x8f7f930?})
time="2025-07-31T11:46:20Z" level=info msg=" Starting pod-lifecycle for Node / Kubelet"
runtime/panic.go:792 +0x132
time="2025-07-31T11:46:20Z" level=info msg=" Starting legacy-storage-invariants for Storage"
time="2025-07-31T11:46:20Z" level=info msg=" Starting legacy-test-framework-invariants for Test Framework"
github.com/openshift/origin/test/extended/util.requiresTestStart(...)time="2025-07-31T11:46:20Z" level=info msg=" Starting staicpod-install-monitor for kube-apiserver"
Member
Author
There was a problem hiding this comment.
While I am figuring out the cause, let see if it is reoccurring.
/test e2e-gcp-ovn-upgrade
Member
Author
There was a problem hiding this comment.
Found in other tests too. For example, https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/30031/pull-ci-openshift-origin-main-e2e-aws-ovn-upgrade/1950870479451459584
Member
Author
There was a problem hiding this comment.
let us see if a957e64 fixes it.
I cannot figure out why the status command stopped panicking after a while because the status cmd seem working and stored in https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/30031/pull-ci-openshift-origin-main-e2e-gcp-ovn-upgrade/1950870499584118784/artifacts/e2e-gcp-ovn-upgrade/openshift-e2e-test/artifacts/junit/adm-upgrade-status/
That means testStarted was handled already, and just a bit late?
Member
Author
There was a problem hiding this comment.
The new run seems fixed.
The latter was failed for other reasons.
/retest-required
Member
Author
|
/test e2e-aws-ovn-upgrade |
Member
Author
|
/retest |
This is to avoid panic caused by `!testsStarted` [1]. [1]. https://github.com/openshift/origin/blob/9307bba172b88036ba0e7eb49a0e2e92947254f9/test/extended/util/test_setup.go#L100-L111
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
xueqzhan
reviewed
Aug 7, 2025
Contributor
xueqzhan
left a comment
xueqzhan
left a comment
There was a problem hiding this comment.
Looking good in general. Just a couple of comments
Contributor
|
/approve |
openshift-ci
bot
added
the
approved
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hongkailiu, wking, xueqzhan The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Member
Author
|
/retest-required |
|
/hold Revision de89f9e was retested 3 times: holding |
openshift-ci
bot
added
the
do-not-merge/hold
Member
Author
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
Member
Author
|
/test e2e-aws-ovn-edge-zones |
Contributor
|
@hongkailiu: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Job Failure Risk Analysis for sha: de89f9e
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: de89f9e
New tests seen in this PR at sha: de89f9e
|
Member
Author
|
/test e2e-gcp-ovn |
Member
|
/override ci/prow/e2e-gcp-ovn |
Contributor
|
@sosiouxme: Overrode contexts on behalf of sosiouxme: ci/prow/e2e-gcp-ovn DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
|
these retries just seem really wasteful when the PR has been through so many already. /override ci/prow/e2e-aws-ovn-edge-zones |
Contributor
|
@sosiouxme: Overrode contexts on behalf of sosiouxme: ci/prow/e2e-aws-ovn-edge-zones, ci/prow/e2e-aws-ovn-fips, ci/prow/e2e-aws-ovn-microshift, ci/prow/e2e-aws-ovn-microshift-serial, ci/prow/e2e-aws-ovn-serial-1of2, ci/prow/e2e-aws-ovn-serial-2of2, ci/prow/e2e-gcp-ovn-upgrade, ci/prow/e2e-metal-ipi-ovn-ipv6, ci/prow/e2e-vsphere-ovn, ci/prow/e2e-vsphere-ovn-upi, ci/prow/okd-scos-images DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-merge-bot
bot
merged commit 7e8f07d
into
openshift:main
36 of 47 checks passed
Contributor
|
[ART PR BUILD NOTIFIER] Distgit: openshift-enterprise-tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pick up the WIP from #29977