Skip to content

CNTRLPLANE-3737: Add find-push-pipelinerun script - #8851

Merged
openshift-merge-bot[bot] merged 6 commits into
openshift:mainfrom
celebdor:find-push-pipelinerun
Jun 30, 2026
Merged

openshift-merge-bot[bot] merged 6 commits into
openshift:mainfrom
celebdor:find-push-pipelinerun

Conversation

@celebdor

@celebdor celebdor commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add hack/tools/scripts/find-push-pipelinerun.sh to find Konflux on-push PipelineRun(s) triggered by a merged GitHub PR
  • Accepts a bare PR number (repo inferred via gh), full GitHub URL, or owner/repo#number format
  • Queries live PipelineRuns via oc get, falls back to KubeArchive REST API for archived runs
  • Displays PipelineRun name, status, creation timestamp, and Konflux UI URL (from pipelinesascode.tekton.dev/log-url annotation)
  • Optional component name filter
  • Includes bats unit tests for input parsing, JSON formatting, and component filtering

Example

$ find-push-pipelinerun.sh 8761
PR https://github.com/openshift/hypershift/pull/8761 merged at commit aeadb45792...

No live PipelineRuns found, querying KubeArchive...
(source: KubeArchive)

NAME                                               STATUS               CREATED                   URL
hypershift-cli-mce-27-on-push-7j48x                Failed               2026-06-26T13:47:51Z      https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/...
hypershift-release-mce-27-on-push-jvq72            Completed            2026-06-26T13:47:51Z      https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/...

Test plan

  • shellcheck passes clean
  • bats unit tests pass (14/14)
  • Manual test: bare PR number, owner/repo#number, full URL
  • Manual test: component filter
  • Manual test: KubeArchive fallback (archived PipelineRuns)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a command-line utility to locate Tekton push PipelineRun records for a merged pull request, resolving PR references from common GitHub URL formats.
    • Outputs results in a readable table (including image and log details) and supports optional component-name filtering.
    • Added live “watch” mode to refresh results until all matching pipeline runs reach a terminal state.
  • Bug Fixes
    • Improved robustness when optional PipelineRun fields (e.g., conditions or log annotations) are missing.
  • Tests
    • Added automated tests covering PR parsing, result formatting (sorting and missing fields), pending-state detection, and component filtering.

celebdor and others added 4 commits June 26, 2026 16:24
Add a helper script to find Konflux on-push PipelineRun(s) triggered
by a merged GitHub PR. Accepts a bare PR number (repo inferred via gh),
a full URL, or owner/repo#number format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PipelineRuns get archived quickly by the kube archiver and are
typically not available via oc get. Query the KubeArchive REST API
as a fallback when no live PipelineRuns are found.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract the pipelinesascode.tekton.dev/log-url annotation from each
PipelineRun to display a clickable link to the Konflux web UI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add unit tests for the pure functions: resolve_pr (input parsing),
format_pipelineruns (JSON to table), and filter_by_component (row
filtering). Add a source guard so the script can be sourced without
running main.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 26, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 26, 2026

Copy link
Copy Markdown

@celebdor: This pull request references CNTRLPLANE-3737 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • Add hack/tools/scripts/find-push-pipelinerun.sh to find Konflux on-push PipelineRun(s) triggered by a merged GitHub PR
  • Accepts a bare PR number (repo inferred via gh), full GitHub URL, or owner/repo#number format
  • Queries live PipelineRuns via oc get, falls back to KubeArchive REST API for archived runs
  • Displays PipelineRun name, status, creation timestamp, and Konflux UI URL (from pipelinesascode.tekton.dev/log-url annotation)
  • Optional component name filter
  • Includes bats unit tests for input parsing, JSON formatting, and component filtering

Example

$ find-push-pipelinerun.sh 8761
PR https://github.com/openshift/hypershift/pull/8761 merged at commit aeadb45792...

No live PipelineRuns found, querying KubeArchive...
(source: KubeArchive)

NAME                                               STATUS               CREATED                   URL
hypershift-cli-mce-27-on-push-7j48x                Failed               2026-06-26T13:47:51Z      https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/...
hypershift-release-mce-27-on-push-jvq72            Completed            2026-06-26T13:47:51Z      https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/...

Test plan

  • shellcheck passes clean
  • bats unit tests pass (14/14)
  • Manual test: bare PR number, owner/repo#number, full URL
  • Manual test: component filter
  • Manual test: KubeArchive fallback (archived PipelineRuns)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7a7cefd3-035b-4752-95fd-9c0f3062d6ff

📥 Commits

Reviewing files that changed from the base of the PR and between 673dc1f and 238ccf1.

📒 Files selected for processing (2)
  • hack/tools/scripts/find-push-pipelinerun.sh
  • hack/tools/scripts/find-push-pipelinerun_test.bats
🚧 Files skipped from review as they are similar to previous changes (2)
  • hack/tools/scripts/find-push-pipelinerun_test.bats
  • hack/tools/scripts/find-push-pipelinerun.sh

📝 Walkthrough

Walkthrough

Adds a Bash script that resolves merged PR references, fetches the merge commit SHA, queries live or archived Tekton PipelineRuns, formats and filters tabular output, and supports watch refreshes. Adds Bats tests for PR parsing, formatting, pending detection, and component filtering.

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant Script
  participant gh
  participant oc
  participant curl
  participant KubeArchive
  participant filter_by_component

  User->>Script: provide PR reference and options
  Script->>gh: resolve PR and fetch mergeCommit.oid
  gh-->>Script: PR metadata
  Script->>oc: query live PipelineRuns by sha and push label
  alt live query fails
    Script->>oc: whoami -t
    oc-->>Script: token
    Script->>curl: request archived PipelineRuns
    curl->>KubeArchive: GET matching PipelineRuns
    KubeArchive-->>curl: archived JSON
    curl-->>Script: archived results
  end
  opt component prefix provided
    Script->>filter_by_component: keep matching TSV rows
    filter_by_component-->>Script: filtered table
  end
  Script->>Script: repeat while pending runs remain
Loading

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The script prints a hardcoded internal KubeArchive hostname in help and query-failure messages, exposing an internal service name. Use a generic placeholder in usage text and omit/redact the host in error messages; keep only env var names or a non-sensitive label.
✅ Passed checks (10 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding the find-push-pipelinerun script.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo-style titles were added; the new Bats tests use static, descriptive names with no run-specific dynamic data.
Test Structure And Quality ✅ Passed PASS: The PR adds Bash/Bats tests only; there is no Ginkgo test code to evaluate for BeforeEach/Eventually/assertion-message patterns.
Topology-Aware Scheduling Compatibility ✅ Passed Only a Bash helper and Bats tests were added; no manifests/controllers or scheduling primitives (affinity, node selectors, PDBs) appear in the changed files.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds only a Bash helper and Bats unit tests; no Ginkgo e2e tests or IPv4/public-network assumptions were added in the test code.
No-Weak-Crypto ✅ Passed The new shell script/test add no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, no custom crypto, and no secret/token comparisons.
Container-Privileges ✅ Passed The PR only adds a shell script and Bats tests; neither contains privileged/hostPID/hostNetwork/hostIPC/SYS_ADMIN/allowPrivilegeEscalation settings, and no K8s manifest was changed.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from Nirshal and bryan-cox June 26, 2026 20:57
@openshift-ci openshift-ci Bot added area/ci-tooling Indicates the PR includes changes for CI or tooling and removed do-not-merge/needs-area labels Jun 26, 2026
@celebdor

Copy link
Copy Markdown
Collaborator Author

/verified by celebdor
Ran the bats based tests and also tried it while verifying @yiraeChristineKim 's PRs for the non-cgo hypershift binary removal

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 26, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@celebdor: This PR has been marked as verified by celebdor.

Details

In response to this:

/verified by celebdor
Ran the bats based tests and also tried it while verifying @yiraeChristineKim 's PRs for the non-cgo hypershift binary removal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

openshift-ci-robot commented Jun 26, 2026

Copy link
Copy Markdown

@celebdor: This pull request references CNTRLPLANE-3737 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • Add hack/tools/scripts/find-push-pipelinerun.sh to find Konflux on-push PipelineRun(s) triggered by a merged GitHub PR
  • Accepts a bare PR number (repo inferred via gh), full GitHub URL, or owner/repo#number format
  • Queries live PipelineRuns via oc get, falls back to KubeArchive REST API for archived runs
  • Displays PipelineRun name, status, creation timestamp, and Konflux UI URL (from pipelinesascode.tekton.dev/log-url annotation)
  • Optional component name filter
  • Includes bats unit tests for input parsing, JSON formatting, and component filtering

Example

$ find-push-pipelinerun.sh 8761
PR https://github.com/openshift/hypershift/pull/8761 merged at commit aeadb45792...

No live PipelineRuns found, querying KubeArchive...
(source: KubeArchive)

NAME                                               STATUS               CREATED                   URL
hypershift-cli-mce-27-on-push-7j48x                Failed               2026-06-26T13:47:51Z      https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/...
hypershift-release-mce-27-on-push-jvq72            Completed            2026-06-26T13:47:51Z      https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/...

Test plan

  • shellcheck passes clean
  • bats unit tests pass (14/14)
  • Manual test: bare PR number, owner/repo#number, full URL
  • Manual test: component filter
  • Manual test: KubeArchive fallback (archived PipelineRuns)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

  • Added a command-line utility to find Tekton push PipelineRun records for a merged pull request.

  • The tool can resolve a PR from common GitHub formats, look up the related commit, and display matching pipeline runs in a readable table.

  • It supports optional filtering by component name and can fall back to archived results when live data is unavailable.

  • Tests

  • Added automated coverage for PR parsing, pipeline run formatting, and component filtering.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@celebdor

Copy link
Copy Markdown
Collaborator Author

/jira refresh

@openshift-ci-robot

openshift-ci-robot commented Jun 26, 2026

Copy link
Copy Markdown

@celebdor: This pull request references CNTRLPLANE-3737 which is a valid jira issue.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.19%. Comparing base (02675db) to head (238ccf1).
⚠️ Report is 59 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8851      +/-   ##
==========================================
+ Coverage   43.01%   43.19%   +0.17%     
==========================================
  Files         766      767       +1     
  Lines       94769    94914     +145     
==========================================
+ Hits        40765    40998     +233     
+ Misses      51185    51052     -133     
- Partials     2819     2864      +45     

see 15 files with indirect coverage changes

Flag Coverage Δ
cmd-support 36.42% <ø> (+<0.01%) ⬆️
cpo-hostedcontrolplane 45.31% <ø> (+0.30%) ⬆️
cpo-other 45.10% <ø> (+0.09%) ⬆️
hypershift-operator 53.53% <ø> (+0.39%) ⬆️
other 31.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
hack/tools/scripts/find-push-pipelinerun.sh (2)

123-124: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Consider URL-encoding the label selector.

The selector embeds , and = directly into the query string. The Kubernetes-style API generally tolerates raw values, but encoding via curl -G --data-urlencode is more robust against future selector values containing reserved characters.

♻️ Suggested approach
-    response="$(curl -sf -H "Authorization: Bearer ${token}" \
-        "${ka_host}/apis/tekton.dev/v1/namespaces/${namespace}/pipelineruns?labelSelector=${selector}")" || {
+    response="$(curl -sf -G -H "Authorization: Bearer ${token}" \
+        --data-urlencode "labelSelector=${selector}" \
+        "${ka_host}/apis/tekton.dev/v1/namespaces/${namespace}/pipelineruns")" || {
         printf "Error: KubeArchive query failed (is %s reachable?)\n" "${ka_host}" >&2
         return 1
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/scripts/find-push-pipelinerun.sh` around lines 123 - 124, The
pipelinerun lookup in find-push-pipelinerun.sh builds the query string with a
raw labelSelector value, which can break if the selector contains reserved
characters. Update the curl call that fetches the Tekton pipelineruns to send
the selector as URL-encoded data instead of interpolating it directly, using the
existing selector variable and the request around the response assignment.

132-145: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

grep "^${component}" treats the filter as a regex.

component is interpreted as a basic regex anchored at line start. For current component names (alphanumeric + hyphen) this is harmless, but a value containing regex metacharacters would match unexpectedly. If literal prefix matching is intended, escape the input or document the regex behavior.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/scripts/find-push-pipelinerun.sh` around lines 132 - 145, The
prefix filter in filter_by_component currently uses grep with "^${component}",
which treats component as a regex instead of a literal string. Update the
matching logic in filter_by_component to either escape the component before
passing it to grep or switch to literal prefix matching, and keep the existing
header/filtered output behavior unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/tools/scripts/find-push-pipelinerun.sh`:
- Around line 61-66: The `gh pr view` call in `find-push-pipelinerun.sh` is
capturing stderr into `pr_json`, which can corrupt the JSON payload before `jq
-r '.state'` parses it. Change the `gh` invocation so stderr is handled
separately from the JSON output, and keep the success path assigned only clean
JSON while preserving the existing failure message behavior in the
`pr_number`/`repo` lookup block.

---

Nitpick comments:
In `@hack/tools/scripts/find-push-pipelinerun.sh`:
- Around line 123-124: The pipelinerun lookup in find-push-pipelinerun.sh builds
the query string with a raw labelSelector value, which can break if the selector
contains reserved characters. Update the curl call that fetches the Tekton
pipelineruns to send the selector as URL-encoded data instead of interpolating
it directly, using the existing selector variable and the request around the
response assignment.
- Around line 132-145: The prefix filter in filter_by_component currently uses
grep with "^${component}", which treats component as a regex instead of a
literal string. Update the matching logic in filter_by_component to either
escape the component before passing it to grep or switch to literal prefix
matching, and keep the existing header/filtered output behavior unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 544a7291-59b1-4bde-8be8-4d041f1ca0be

📥 Commits

Reviewing files that changed from the base of the PR and between c180b15 and 43c0f89.

📒 Files selected for processing (2)
  • hack/tools/scripts/find-push-pipelinerun.sh
  • hack/tools/scripts/find-push-pipelinerun_test.bats

Comment thread hack/tools/scripts/find-push-pipelinerun.sh Outdated
- Add -w/--watch flag to poll until all PipelineRuns complete
- Show IMAGE_URL@IMAGE_DIGEST for completed PipelineRuns
- Fix readonly re-sourcing issue for testability
- Fix gh stderr being captured into JSON parsing
- Move set -euo pipefail into main() to avoid affecting sourcing shell
- Add has_pending tests and image output tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Jun 26, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/tools/scripts/find-push-pipelinerun.sh`:
- Around line 203-205: The missing-PR-argument path in find-push-pipelinerun.sh
currently returns success after calling usage, which makes callers think the
lookup completed; update the argument validation branch in the main
positional-argument handling so it returns a non-zero status when positional is
empty, while keeping the explicit help path in usage/--help behavior at zero.
Use the existing usage function and the positional array check to locate the
fix, and ensure only the missing required argument case fails.
- Around line 141-143: Add bounded timeouts to the KubeArchive fetch in
find-push-pipelinerun.sh so the curl call cannot hang forever on a stalled API
connection. Update the request that builds response from
`${ka_host}/apis/tekton.dev/v1/namespaces/${namespace}/pipelineruns` to include
both a connect timeout and an overall timeout, and keep the existing
fallback/error handling so the lookup and --watch refresh fail fast instead of
blocking indefinitely.
- Around line 90-97: The jq expression in format_pipelineruns() is malformed
because the fallback is applied in a way that prevents parsing and causes jq to
exit before rendering the table. Update the image construction in
find-push-pipelinerun.sh so the default empty string is applied inside the image
value or directly on the IMAGE_URL lookup, keeping the IMAGE_DIGEST append logic
intact and ensuring the expression parses correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c630d181-1d1d-4a38-8be6-b20c0ccc0750

📥 Commits

Reviewing files that changed from the base of the PR and between 43c0f89 and 673dc1f.

📒 Files selected for processing (2)
  • hack/tools/scripts/find-push-pipelinerun.sh
  • hack/tools/scripts/find-push-pipelinerun_test.bats

Comment thread hack/tools/scripts/find-push-pipelinerun.sh Outdated
Comment thread hack/tools/scripts/find-push-pipelinerun.sh Outdated
Comment thread hack/tools/scripts/find-push-pipelinerun.sh Outdated

@jparrill jparrill left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean dev tool — well-decomposed functions, good error handling, 14 bats tests covering the key paths. LGTM.

/approve

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 29, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@openshift-ci

openshift-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: celebdor, jparrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 29, 2026
@cwbotbot

cwbotbot commented Jun 29, 2026

Copy link
Copy Markdown

Test Results

e2e-aws

e2e-aks

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2071570560260771840 | Cost: $2.1204725000000004 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@celebdor
celebdor force-pushed the find-push-pipelinerun branch from 673dc1f to 238ccf1 Compare June 30, 2026 09:10
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2026
@celebdor

Copy link
Copy Markdown
Collaborator Author

/override ci/prow/e2e-aks
/override ci/prow/e2e-aws
/override ci/prow/e2e-aws-upgrade-hypershift-operator
/override ci/prow/e2e-azure-v2-self-managed
/override ci/prow/e2e-kubevirt-aws-ovn-reduced
/override ci/prow/e2e-v2-aws
/override ci/prow/e2e-v2-gke

@openshift-ci

openshift-ci Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-upgrade-hypershift-operator, ci/prow/e2e-azure-v2-self-managed, ci/prow/e2e-kubevirt-aws-ovn-reduced, ci/prow/e2e-v2-aws, ci/prow/e2e-v2-gke

Details

In response to this:

/override ci/prow/e2e-aks
/override ci/prow/e2e-aws
/override ci/prow/e2e-aws-upgrade-hypershift-operator
/override ci/prow/e2e-azure-v2-self-managed
/override ci/prow/e2e-kubevirt-aws-ovn-reduced
/override ci/prow/e2e-v2-aws
/override ci/prow/e2e-v2-gke

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

I now have all the evidence. Here is the complete analysis:

Test Failure Analysis Complete

Job Information

  • Prow Job: pull-ci-openshift-hypershift-main-okd-scos-images
  • Build ID: 2071884010002321408
  • Target: [images] (variant: okd-scos)
  • Cluster: build01
  • PR: openshift/hypershift#8851CNTRLPLANE-3737: Add find-push-pipelinerun script
  • Job State: error
  • Duration: 30m (09:10:16Z → 09:40:16Z, never started)

Test Failure Analysis

Error

Pod scheduling timeout.

0/61 nodes are available: 13 node(s) didn't match Pod's node affinity/selector,
2 Insufficient memory, 46 node(s) had untolerated taint(s).
no new claims to deallocate, preemption: 0/61 nodes are available:
2 Insufficient memory, 59 Preemption is not helpful for scheduling.

Summary

The okd-scos-images CI job failed before any test code executed. The ci-operator pod could not be scheduled onto any node in the build01 cluster within the 30-minute timeout. Of 61 available nodes, 46 had untolerated taints, 13 didn't match the pod's node affinity/selector (requiring kubernetes.io/arch in [amd64, arm64] and excluding hostname ip-10-0-155-94.ec2.internal), and the remaining 2 had insufficient memory. This is a CI infrastructure capacity issue — completely unrelated to PR #8851's code changes (which add a find-push-pipelinerun script). The PR only touches shell scripts under hack/ and does not modify any Dockerfiles, CI configuration, or pod resource requests.

Root Cause

The root cause is CI infrastructure resource exhaustion on the build01 cluster at the time the job was dispatched (2026-06-30 09:10 UTC).

The pod was configured by the multi-arch scheduling system (labels show multiarch.openshift.io/multi-arch, multiarch.openshift.io/node-affinity: set) with the following constraints:

  1. Required node affinity: kubernetes.io/arch must be amd64 or arm64 — this excluded 13 nodes (likely nodes with other architectures like s390x or ppc64le).
  2. Toleration for node-role.kubernetes.io/ci-prowjobs-worker — but 46 nodes had other taints not tolerated by this pod, leaving them unavailable.
  3. Pod anti-affinity: required anti-affinity against pods with label pod-scaler.openshift.io/measured=false, which may have further constrained placement.
  4. Memory: The 2 remaining candidate nodes had insufficient memory to schedule even the modest resource request (62 MiB for the test container + 250 MiB for the sidecar).

The pod remained in Pending phase for 30 minutes until Prow's pod scheduling timeout killed it. No build log, no JUnit XML, and no artifacts were produced — only prowjob.json, podinfo.json, started.json, and finished.json exist.

This is a transient CI infrastructure issue. The build01 cluster was at capacity with insufficient schedulable nodes matching the pod's constraints at that specific time.

Recommendations
  1. Retry the job — Run /test okd-scos-images on the PR. This is a transient scheduling failure unrelated to code changes, and the job should succeed when cluster capacity frees up.

  2. No code changes needed — PR CNTRLPLANE-3737: Add find-push-pipelinerun script #8851 only adds a find-push-pipelinerun shell script under hack/. It does not modify Dockerfiles, CI configuration (openshift-hypershift-main-okd-scos.yaml), pod resource requests, or node affinity rules.

  3. If the retry also fails with the same scheduling error, escalate to the CI infrastructure team (#forum-ocp-testplatform on Slack) as it may indicate a sustained capacity issue on the build01 cluster.

Evidence
Evidence Detail
Job state error — pod never started executing
Pod phase Pending for entire 30-minute lifetime
Pod condition PodScheduled: False, reason: Unschedulable
Scheduling message 0/61 nodes are available: 13 node(s) didn't match Pod's node affinity/selector, 2 Insufficient memory, 46 node(s) had untolerated taint(s)
Pod created 2026-06-30T09:10:16Z
Pod deleted (timeout) 2026-06-30T09:40:16Z (exactly 30m later)
Target cluster build01
Node assigned None — pod was never scheduled
Build log Does not exist — pod never ran
JUnit XML Does not exist — pod never ran
Node affinity Required: arch in [amd64, arm64], excluded hostname ip-10-0-155-94.ec2.internal
Pod anti-affinity Required: avoid nodes with pod-scaler.openshift.io/measured=false pods
Toleration node-role.kubernetes.io/ci-prowjobs-worker (NoSchedule)
Multi-arch labels multiarch.openshift.io/multi-arch, amd64, arm64 — scheduling gate removed
Resource requests test: 15m CPU / 62 MiB memory; sidecar: 150m CPU / 250 MiB memory

@celebdor
celebdor requested a review from dhgautam99 June 30, 2026 15:30
Comment on lines +4 to +8
_set_default() { declare -g -r "$1"="$2" 2>/dev/null || true; }
_set_default DEFAULT_NAMESPACE "crt-redhat-acm-tenant"
_set_default DEFAULT_KA_HOST "https://kubearchive-api-server-product-kubearchive.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com"
_set_default LOG_URL_ANNOTATION "pipelinesascode.tekton.dev/log-url"
_set_default WATCH_INTERVAL 15

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script fails when bash version is less than 4.2. For better portability purposes, it can be replaced using below declaration:
I tested it and it worked well on Mac as well.

Suggested change
_set_default() { declare -g -r "$1"="$2" 2>/dev/null || true; }
_set_default DEFAULT_NAMESPACE "crt-redhat-acm-tenant"
_set_default DEFAULT_KA_HOST "https://kubearchive-api-server-product-kubearchive.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com"
_set_default LOG_URL_ANNOTATION "pipelinesascode.tekton.dev/log-url"
_set_default WATCH_INTERVAL 15
: "${DEFAULT_NAMESPACE:=crt-redhat-acm-tenant}"
: "${DEFAULT_KA_HOST:=https://kubearchive-api-server-product-kubearchive.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com}"
: "${LOG_URL_ANNOTATION:=pipelinesascode.tekton.dev/log-url}"
: "${WATCH_INTERVAL:=15}"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to keep the readonly protection from more modern bash. I recommend using mac os brew to upgrade your bash

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then, current config looks ok.

@celebdor

Copy link
Copy Markdown
Collaborator Author

/override "ci/prow/okd-scos-images"

@openshift-ci

openshift-ci Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

@celebdor: Overrode contexts on behalf of celebdor: ci/prow/okd-scos-images

Details

In response to this:

/override "ci/prow/okd-scos-images"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@dhgautam99

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@celebdor

Copy link
Copy Markdown
Collaborator Author

/verified by dhgautam99

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 30, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@celebdor: This PR has been marked as verified by dhgautam99.

Details

In response to this:

/verified by dhgautam99

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

@celebdor: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit fea363d into openshift:main Jun 30, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ci-tooling Indicates the PR includes changes for CI or tooling jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants