Skip to content

chore: sets the initial config for a market place and a hello world plugin - #1

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift-eng:mainfrom
enxebre:marketplace-seed
Oct 16, 2025
Merged

openshift-merge-bot[bot] merged 1 commit into
openshift-eng:mainfrom
enxebre:marketplace-seed

Conversation

@enxebre
enxebre force-pushed the marketplace-seed branch 2 times, most recently from 513a3ba to 2516425 Compare October 14, 2025 07:58
Comment thread plugins/hello-world/commands/echo.md Outdated
description: Hello world plugin implementation
---

# Hello Command

@brandisher brandisher Oct 14, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In relation to this comment, what do you think about the this command being the definition for what new commands should look like?

I understand the purpose is to seed the marketplace but we could accomplish two goals with one PR.

@enxebre enxebre Oct 14, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this idea. I updated it to serve as a ref implementation

@enxebre enxebre closed this Oct 14, 2025
@enxebre enxebre reopened this Oct 14, 2025
@stbenjam

Copy link
Copy Markdown
Member

/label lgtm
/label approved

@openshift-ci

openshift-ci Bot commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: The label(s) /label lgtm cannot be applied. These labels are supported: acknowledge-critical-fixes-only, platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, ux-approved, no-qe, downstream-change-needed, rebase/manual, cluster-config-api-changed, run-integration-tests, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/valid-bug, ok-to-test, stability-fix-approved, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

Details

In response to this:

/label lgtm
/label approved

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@enxebre

enxebre commented Oct 14, 2025

Copy link
Copy Markdown
Contributor Author

ptal @brandisher

@bryan-cox

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

@bryan-cox: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@brandisher

Copy link
Copy Markdown

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

@brandisher: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 15, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kuiwang02 added a commit to kuiwang02/ai-helpers that referenced this pull request Oct 16, 2025
Follow the section structure from PR openshift-eng#1:
- ## Name
- ## Synopsis
- ## Description
- ## Implementation
- ## Return Value
- ## Examples
- ## Arguments

This addresses review feedback to adhere to the established documentation format.
@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci

openshift-ci Bot commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

@stbenjam: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@stbenjam

Copy link
Copy Markdown
Member

/approve

@stbenjam

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 16, 2025
@openshift-ci

openshift-ci Bot commented Oct 16, 2025

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brandisher, bryan-cox, enxebre, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [brandisher,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 16, 2025
@openshift-merge-bot
openshift-merge-bot Bot merged commit 3066b71 into openshift-eng:main Oct 16, 2025
1 check passed
mpatlasov added a commit to mpatlasov/ai-helpers that referenced this pull request Nov 7, 2025
Let AI to apply changes manually for conflicted files.

Example: in the repo https://github.com/openshift/csi-operator.git , there is a commit (`9fa7ded32d45e71eb6715855ffc7f34852f375de`) in the branch `release-4.17`. The command `patch` failed in the branch `release-4.16`:

```
$ git show 9fa7ded32d45e71eb6715855ffc7f34852f375de | patch -p1 --no-backup-if-mismatch
...
patching file assets/overlays/azure-disk/generated/hypershift/manifests.yaml
Hunk openshift-eng#1 FAILED at 13.
```

However, Claude is smart enough to resolve the conflict manually. The result of imporved cherry-pick-by-patch is here: openshift/csi-operator@b236f6c
mpatlasov added a commit to mpatlasov/ai-helpers that referenced this pull request Nov 7, 2025
Let AI to apply changes manually for conflicted files.

Example: in the repo https://github.com/openshift/csi-operator.git , there is a commit (`9fa7ded32d45e71eb6715855ffc7f34852f375de`) in the branch `release-4.17`. The command `patch` failed in the branch `release-4.16`:

```
$ git show 9fa7ded32d45e71eb6715855ffc7f34852f375de | patch -p1 --no-backup-if-mismatch
...
patching file assets/overlays/azure-disk/generated/hypershift/manifests.yaml
Hunk openshift-eng#1 FAILED at 13.
```

However, AI assistant is smart enough to resolve the conflict manually. The result of imporved cherry-pick-by-patch is here: openshift/csi-operator@b236f6c
mpatlasov added a commit to mpatlasov/ai-helpers that referenced this pull request Mar 11, 2026
Let AI to apply changes manually for conflicted files.

Example: in the repo https://github.com/openshift/csi-operator.git , there is a commit (`9fa7ded32d45e71eb6715855ffc7f34852f375de`) in the branch `release-4.17`. The command `patch` failed in the branch `release-4.16`:

```
$ git show 9fa7ded32d45e71eb6715855ffc7f34852f375de | patch -p1 --no-backup-if-mismatch
...
patching file assets/overlays/azure-disk/generated/hypershift/manifests.yaml
Hunk openshift-eng#1 FAILED at 13.
```

However, AI assistant is smart enough to resolve the conflict manually. The result of imporved cherry-pick-by-patch is here: openshift/csi-operator@b236f6c
dfarrell07 added a commit to dfarrell07/ai-helpers that referenced this pull request May 29, 2026
Blind test openshift-eng#1 found two critical issues:

1. Agent missed 6 test files when adding AtomicFIFO gate disabling
   because it only found SetFromMap-based files, not os.Setenv/t.Setenv
   ones. Add explicit grep command to find ALL files with existing
   gate disabling before adding new gates.

2. Agent migrated x/exp imports to stdlib but didn't run go mod tidy
   to remove x/exp from go.mod/vendor. Add explicit container-based
   cleanup instruction since local Go may be too old.

Also fix t.Fatalf in init() example (t not available in init).

Signed-off-by: Daniel Farrell <dfarrell@redhat.com>
redhat-chai-bot added a commit to redhat-chai-bot/ai-helpers that referenced this pull request Jul 2, 2026
… commands

Addresses review feedback on the prow-job-analysis megaskill PR.

BLOCKING openshift-eng#1 — dangling skill references:
- Delete thin-wrapper commands analyze-prow-job-test-failure,
  analyze-prow-job-install-failure, extract-prow-job-must-gather
  (superseded by the prow-job-analysis skill; per review)
- Repoint every reference to the 5 removed skills
  (prow-job-analyze-{test,install,metal-install}-failure,
  prow-job-artifact-search, prow-job-extract-must-gather) to
  ci:prow-job-analysis across commands, skills, references, README,
  PLUGINS.md, and the openshift/teams plugins
- Fix the payload-analysis quality gate to match the single-skill model
- Point analyze-disruption at the relocated prow_job_artifact_search.py

BLOCKING openshift-eng#2 — must-gather ghost references:
- Remove the no-op --fast flag (must-gather is no longer auto-extracted)
- Correct the "renamed to content/ by extraction script" claims: the
  hash directory is preserved after tar -xf, not renamed
- Mark the must-gather-analyzer scripts as run manually from the
  must-gather plugin (networking, resource-exhaustion, hypershift)
- Add a "Must-Gather Download & Extraction" section (fetch tar,
  tar -xf, decompress nested archives) to the artifacts reference

Bump ci 0.0.56->0.0.57 and openshift 0.0.6->0.0.7; sync marketplace.json
and regenerate docs. make lint passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
openshift-merge-bot Bot pushed a commit that referenced this pull request Jul 8, 2026
… (supersedes #597) (#604)

* feat(ci): add prow-job-analysis megaskill, remove 5 superseded skills

Add the prow-job-analysis megaskill (SKILL.md plus reference guides for
artifacts, aggregated jobs, CI-infrastructure changes, cloud-provider
errors, disruption, flaky tests, HyperShift, install general/metal,
networking, resource exhaustion, test-extension binaries, and upgrades).

Remove the five skills it supersedes and relocate the artifact-search
helper script into the megaskill:
- prow-job-analyze-test-failure
- prow-job-artifact-search (prow_job_artifact_search.py moved into the megaskill)
- prow-job-analyze-install-failure
- prow-job-analyze-metal-install-failure
- prow-job-extract-must-gather

Resolve CodeRabbit review feedback:
- Tag every fenced code block with a language (MD040)
- artifacts.md / ci-infrastructure-changes.md: {JOB_NAME} is the top-level
  bucket path only; step artifacts stay under artifacts/{target}/
- SKILL.md: align URL examples with the artifact-search parser contract (logs/)
- networking.md: escape the pipe in the ICSP/IDMS table cell
- upgrade.md: route bare-metal install failures to install/metal.md
- hypershift.md: route Go-test/CI-step failures to build-log/step-graph
  analysis instead of JUnit XML / the extension-binary guide
- aggregated.md: fix the contradictory "all children passed" failure mode
- cloud-provider-errors.md: fix the Region/Zone Availability anchor links

Bump the ci plugin to 0.0.56 and regenerate marketplace.json + docs/index.html.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): correct accuracy/cross-ref issues in prow-job-analysis review

Self-review of the prow-job-analysis megaskill. Fixes verified against
upstream sources (dev-scripts Makefile, baremetalds gather step,
openshift-tests-extension types.go) and repo-internal grep.

Accuracy (verified against source):
- metal.md: swap dev-scripts script numbers — 03=build_installer,
  04=setup_ironic (were reversed) across table, failure patterns,
  artifact map, quick-ref
- metal.md: metal gather archives are gzipped — libvirt-logs.tar.gz,
  log-bundle-*.tar.gz, squid-logs-*.tar.gz; fix $-anchored greps that
  missed them; add installer-status.txt; add SNO bootstrap-in-place note;
  replace invented OFCIR build-log JSON/pool schema with grep guidance
- test-extension-binaries.md: correct OTE JSON — result/passed (not
  status/pass), duration int64 ns, component type, snake_case source
  fields, apiVersion v1.1; replace rot-prone 34-row registry catalog
  with naming pattern + discovery command + verified exceptions
- ci-infrastructure-changes.md / cloud-provider-errors.md: fix Boskos
  "(legacy)" and OFCIR mislabelling; OFCIR = OpenShift Fleeting CI
  Resources, consistent across files; Boskos brokers cloud quota slices
- general.md: log bundles are gzipped (robust .tar(.gz)? matching)
- upgrade.md: remove fabricated `drain exceeded timeout`/`5400s` MCD
  strings and false 90-min default; fix jq-on-YAML must-gather bugs
  (use yq / oc_cmds text); add EUS-to-EUS paused-MCP note
- resource-exhaustion.md: fix gather-extra/oc_cmds path (missing
  /artifacts/); SystemOOM (not OOMKilling) for kubelet events
- disruption.md: fix self-inconsistent interval JSON example

Cross-references (dead-ends found via grep):
- flaky-test-identification.md: drop dead importing_release→networking
  route; add HyperShift onward route; add utilizing_cluster_claim
- hypershift.md: konnectivity is HCP-specific (networking.md lacks it);
  correct See Also descriptions to what siblings actually cover
- networking.md: retarget OVS-stall ref to disruption.md; move
  misfiled Hairpinning bullet from DNS to Service Connectivity

Progressive disclosure / conciseness:
- general.md: drop duplicated per-mode procedure block
- metal.md: point to artifacts.md for canonical metal layout
- SKILL.md: add mkdir -p so download commands work

make lint: 0 errors, 0 warnings, Grade A+, all checks passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): repoint dangling refs to removed skills, delete thin-wrapper commands

Addresses review feedback on the prow-job-analysis megaskill PR.

BLOCKING #1 — dangling skill references:
- Delete thin-wrapper commands analyze-prow-job-test-failure,
  analyze-prow-job-install-failure, extract-prow-job-must-gather
  (superseded by the prow-job-analysis skill; per review)
- Repoint every reference to the 5 removed skills
  (prow-job-analyze-{test,install,metal-install}-failure,
  prow-job-artifact-search, prow-job-extract-must-gather) to
  ci:prow-job-analysis across commands, skills, references, README,
  PLUGINS.md, and the openshift/teams plugins
- Fix the payload-analysis quality gate to match the single-skill model
- Point analyze-disruption at the relocated prow_job_artifact_search.py

BLOCKING #2 — must-gather ghost references:
- Remove the no-op --fast flag (must-gather is no longer auto-extracted)
- Correct the "renamed to content/ by extraction script" claims: the
  hash directory is preserved after tar -xf, not renamed
- Mark the must-gather-analyzer scripts as run manually from the
  must-gather plugin (networking, resource-exhaustion, hypershift)
- Add a "Must-Gather Download & Extraction" section (fetch tar,
  tar -xf, decompress nested archives) to the artifacts reference

Bump ci 0.0.56->0.0.57 and openshift 0.0.6->0.0.7; sync marketplace.json
and regenerate docs. make lint passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(ci): apply terseness cuts to prow-job-analysis references

Remove verified padding (restatement, generic reasoning, redundant
checklists) from five reference files while preserving all domain
strings, error messages, tables, grep patterns, and Quick References.

- ci-infrastructure-changes.md: add a Table of Contents; collapse the
  third-pass "CI Job Lifecycle" phase breakdown to the key rule; drop
  the "Common CI Infrastructure Failure Patterns" re-describe (folding
  the new Step Script Bug guidance into the earlier indicators); cut the
  closing classification checklist.
- install/metal.md: delete "Tips and Best Practices" (all restatements);
  collapse grep cookbooks to keyword lists; keep one download recipe.
- upgrade.md: replace "Putting It All Together" with a compact ordered
  list pointing at the Key Files table (grep markers + parser command
  preserved); trim the 4x-repeated ClusterOperator JSON to its
  humanMessage patterns.
- aggregated.md: keep one parent-build-log warning; cut the rationale
  essay and self-evident glossary; compress "Multi-Step vs True
  Aggregated" (defer step-registry to ci-infrastructure-changes.md);
  drop the duplicate checklist.
- disruption.md: compress the disruption-causation checklist; drop
  Scenarios A/B/C (folding C's nugget into the upgrade-phase list); cut
  "Why Cross-Run Analysis Matters".

Cross-file dedup:
- parse_disruption.py docs now link to analyze-disruption/SKILL.md.
- artifacts.md interval format links to disruption.md (path row kept).
- job_labels download commands owned by artifacts.md; others link to it.

Net -569 lines. Bump ci 0.0.57->0.0.58; sync marketplace.json and
regenerate docs. make lint passes (0 errors, 0 warnings, A+).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(ci): add test-failure root-cause reference; close routing dead-end

flaky-test-identification.md is the "start here" for any failing test, but
once it classified a failure as a real product regression in a plain
(non-extension, non-install, non-upgrade) e2e test, its onward routes went
nowhere — the deleted prow-job-analyze-test-failure skill's core methodology
had no home.

Add references/test-failure.md owning plain e2e regression root-cause:
- locate test source from the Go/Ginkgo test name (openshift/origin
  test/extended, vendored k8s conformance, extension codeLocations)
- read JUnit failure message + stack-trace shape
- scope namespace/component from the [sig-*] prefix
- pin the failure window from interval files and temporally correlate
  cluster/operator events
- trace crash-looping containers via pod containerStatuses
  (exitCode 137/143/1/2, reason), previous.log, dependency chain
- "never stop at high-level symptoms" synthesis
- route to networking/resource-exhaustion/disruption/cloud references when
  the root cause lands in another domain

Wire it into the routing tables:
- SKILL.md failure routing table
- flaky-test-identification.md: product-regression class, "use a different
  reference" line, JUnit product-behavior sentence, triage-checklist hand-off,
  and See Also

Bump ci 0.0.58 -> 0.0.59; sync marketplace.json and regenerate docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(ci): add operating-system-changes (RHCOS) reference to prow-job-analysis

Add references/operating-system-changes.md covering RHCOS-related failure
debugging, as requested in PR review: when and why to start debugging
OS-level issues (odd cri-o/container-start behavior, NetworkManager
networking problems, kernel panics, systemd/SELinux, storage/rpm-ostree).

The reference explains RHCOS as the immutable, rpm-ostree-managed OS
shipped as a payload component (rhel-coreos / rhel-coreos-10), how an OS
change propagates via the MCO/MCD (drain -> deploy -> reboot), and the
highest-signal ways to suspect the OS layer: RHCOS variant isolation
(RHCOS 9 vs 10) and payload-boundary blast radius. For payloads it points
at the deep RPM-level changelog available in the snapshot (release
controller nodeImageStreams, summary.json rhcos_changes[], rpmdb.sqlite),
and notes RHCOS RPM changes are suspects for the platform team, not
revert candidates. Includes failure-pattern and artifact-location tables
(journals, host_service_logs, serial console, failed-units, MCO logs) and
a quick triage checklist. Deep MCO drain mechanics stay owned by
upgrade.md to avoid duplication.

Wire it into SKILL.md: a Failure Routing Table row and a job-name
classification row (rhcos9/rhcos10/rhcos9_10/rt).

Bump ci 0.0.59->0.0.60; sync marketplace.json and regenerate docs.
make lint passes (0 errors, 0 warnings, A+).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(ci): unify payload-analysis subagent routing to prow-job-analysis

The payload-analysis subagent prompt split failed-job investigation into
"install failure" vs "test failure" branches, each pointing at
`ci:prow-job-analysis` separately, and pre-classified the failure via
JUnit before choosing. That routing is redundant: prow-job-analysis
already identifies the job type, classifies the failure, and routes to
the specialized reference (install, metal, test, upgrade, etc.)
internally.

- Replace the install/test bifurcation with one imperative directive to
  use `ci:prow-job-analysis` for every failed job.
- Drop the "First, check the JUnit results..." pre-classification step;
  that is prow-job-analysis's job now.
- Keep the trace-to-root-cause and infrastructure-flake guidance
  (analysis quality, not routing).

Also delete docs/data.json, a stale orphaned artifact removed during the
skillsaw migration (9dd2118) but later resurrected. Nothing regenerates
it (skillsaw docs writes only index.html) and nothing references it
(index.html fetches live data). It still listed the four deleted
prow-job-analyze-*/artifact-search/extract-must-gather skills -- the only
remaining dangling references to them in the repo.

Bump ci plugin 0.0.60 -> 0.0.61 and sync marketplace/docs via `make update`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: address stbenjam & coderabbit review on prow-job-analysis references

Reference-file corrections from PR #597 review:

- operating-system-changes.md: reword "Use when" opener as a diagnosis of
  exclusion (you don't know a failure is OS-level until other modes are ruled
  out) instead of asserting the origin up front.
- install/general.md & install/metal.md: make the imperative "Use when..."
  the first prose line under the title.
- install/general.md: "also see metal.md" -> "you must read metal.md";
  add major-upgrade (4->5) case; clarify {bucket-path} is the specific job
  run's path (going up a level lists tens of thousands of jobs); note log
  bundles are stored decompressed (never .tar.gz) and are now uploaded
  exploded as a directory; defer must-gather detail to the artifacts
  reference.
- artifacts.md: aggregated JUnit path uses {underlying-job-name}.
- test-extension-binaries.md: step-log path uses {target}, not {JOB_NAME}.

Bump ci plugin 0.0.61 -> 0.0.62 and regenerate docs/marketplace.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ci): incorporate not-stbenjam eval review into payload-analysis

Applies 8 generalized behavioral improvements to the payload-analysis
SKILL.md to reduce wrong-PR attribution and confidence inflation:

1. Conflict-adjudication protocol (Step 5b) — contradictory root causes
   for the same failure signature are UNRESOLVED, not vibe-broken ties;
   require discriminating evidence from the exact failing operation,
   "cleared" needs positive evidence from the failing code path, absent
   log lines aren't evidence under truncation, causal chains must be
   shown to execute, and overrides update the stored per-job root cause.
2. Rubric de-duplication and tiering (Step 6.1) — fold the redundant
   "single candidate" signal into component exclusivity; tier
   error-message match (+40 direct / +20-30 same code path / +10 same
   subsystem); require an itemized per-signal breakdown; re-verify
   top-tier claims when the raw sum exceeds the cap.
3. Tighten force-accept (Step 6.4) — moot when phase is already
   Accepted; define "temporary" via self-resolve-without-human-action.
4. Score every distinct failure mode, not just the dominant one;
   "intermittent/flake" is a conclusion requiring evidence.
5. Causal CI-infrastructure changes must appear as scored candidates[]
   entries even when failure_type is infra (Steps 3.6 and 6.5).
6. Infrastructure exclusion — unrelated PRs score at/near zero for
   affirmatively infra-caused failures.
7. Surface payload-chain context (consecutive rejections, last accepted
   payload, per-job persistence) in the executive summary (Step 7.1).
8. Final mechanical self-check and verbatim phase recording (Steps 3.1,
   Step 10).

Bumps ci plugin 0.0.62 -> 0.0.63; syncs marketplace.json and docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): anchor payload-analysis output dir, tighten scoring; finish stbenjam reference fixes

payload-analysis SKILL.md:
- Step 1 captures OUTPUT_DIR="$(pwd)" before anything else; the results
  YAML (6.5), HTML summary (7), and autodl JSON (8) are all written
  under $OUTPUT_DIR.
- Step 10 self-check verifies the three exact spec'd filenames at
  $OUTPUT_DIR (no glob) and first confirms all Step 4 subagents and the
  Step 9 reviewer completed before assembling the report.
- Step 6.1: recorded confidence score MUST equal min(100, sum of
  itemized signals), each signal at its defined weight with one line of
  evidence; enumerate every failure mode per job before scoring and give
  each its own rubric pass; flake dismissals must cite evidence.
- Step 6.1: when a job has zero causally-linked candidates, require a
  per-job rationale and an explicit cross-job correlation statement.

prow-job-analysis references:
- test-failure.md: drop the unhelpful emphasis asterisks and trim the
  verbose opener (stbenjam).
- operating-system-changes.md: reword trigger to a diagnosis of
  exclusion ("Use when no other failure mode explains the cause...").
- upgrade.md: open with an imperative "Use when" trigger as the first
  prose line and fold the redundant "When to Use" list into it.

Version bump ci 0.0.63 -> 0.0.64 via make update (marketplace.json + docs).
make lint: A+, 0 errors, 0 warnings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ci): add prow-job-analysis eval with 10 failure-mode cases

Adds a skillsaw eval for the ci:prow-job-analysis skill, modeled on
eval-payload-analysis.yaml.

- eval-prow-job-analysis.yaml: mode=case eval that passes each case's
  prow_job_url to ci:prow-job-analysis and has the runner write a
  prow-job-analysis-report.md. Judges: two deterministic (output_file_exists,
  required_skill_invocations) and two LLM rubrics (analysis_quality 1-5,
  failure_classification_accuracy 1-5), each with a threshold. Dataset points
  at cases/prow-job-analysis/.
- scripts/setup-prow-job-analysis.sh: verifies gcloud/python3/jq, confirms the
  public test-platform-results bucket is reachable, prepares the working dir
  (no snapshot needed — artifacts are read live).
- 10 case dirs, each with input.yaml (plausible Prow URL) and annotations.yaml
  (expected_failure_type, expected_references, expected_root_cause_category,
  notes): metal install/Ironic, aggregated mixed children, upgrade MCO drain,
  hypershift dual-cluster, CI step-registry infra change, extension-binary
  crash, cloud quota exhaustion, disconnected proxy pull, upgrade API
  disruption, and SNO OOM. Spread: install x3, upgrade x2, test x3, infra x2.

Bump ci plugin 0.0.64 -> 0.0.65 and sync marketplace/docs. make lint: A+.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): make prow-job-analysis artifact access work without gcloud

The eval-prow-job-analysis job failed because setup-prow-job-analysis.sh
required the gcloud CLI (which is not installed in the CI eval container)
and prow_job_artifact_search.py used `gcloud storage` for all GCS access.

The test-platform-results bucket is public, so no auth or SDK is needed.
Add a Python-stdlib (urllib) fallback to the public GCS JSON/download API,
used automatically whenever gcloud is unavailable:

- prow_job_artifact_search.py: list/search/fetch fall back to the public
  GCS HTTP API when gcloud is absent (or PROW_ARTIFACT_SEARCH_NO_GCLOUD=1).
  JSON output is identical to the gcloud path. Verified against gcloud on a
  real job (search **/*.json 227==227, **/*.tar 8==8, **/nodes 1==1, list
  matches the GCS API exactly); fetch reports the decompressed size for
  gzip-transcoded artifacts.
- setup-prow-job-analysis.sh: gcloud is now optional (python3/jq required);
  bucket reachability and pre-warm use the public API / bundled script, so
  the script no longer exits 2 when gcloud is missing.
- Bump ci plugin 0.0.65 -> 0.0.66; sync marketplace.json and regen docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(ci): replace hallucinated prow-job-analysis eval cases with real jobs

The 10 prow-job-analysis eval cases used fabricated sequential build IDs
(1780146820004390401-410) that do not exist in GCS. Replace all of them
with real, failed CI jobs whose artifacts are present in the public
test-platform-results bucket (each verified: build-log.txt returns HTTP
200 and finished.json reports result=FAILURE).

Cases now cover: metal IPI install (EPEL mirror outage), aggregated mixed
children (disruption threshold), two-node-arbiter upgrade operator
degradation, HyperShift release-image import failure, CI build-cluster
scheduling exhaustion, MCO/MachineConfigPool timeout, GCP NETWORKS quota
exhaustion, disconnected payload import, multi-arch upgrade API disruption,
and GCP node-not-ready timeout. Each input.yaml carries a real
prow_job_url and each annotations.yaml sets expected_failure_type,
expected_references, expected_root_cause_category, and notes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(images): add gcloud CLI to container image

Install the google-cloud-cli package in both images/Dockerfile and
images/Dockerfile.nested-podman so gcloud (and the bundled gsutil and
bq binaries) are available in the image. This is needed for accessing
public GCS artifacts during Prow job analysis via `gcloud storage` and
`gsutil`.

Follows the existing external-repo pattern: adds
images/repos/google-cloud-cli.repo plus a bundled GPG key at
images/repos/RPM-GPG-KEY-google-cloud. The key file contains BOTH of
Google's signing keys as separate armor blocks (like RPM-GPG-KEY-gh-cli):
the RPM package-signing key (used by gpgcheck) and the Artifact Registry
metadata-signing key (used by repo_gpgcheck), so both gpgcheck=1 and
repo_gpgcheck=1 succeed. The new .repo and key files are picked up
automatically by the existing COPY globs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ci): enrich prow-job-analysis eval schema + verified case-001/002 fixtures

Add structured ground-truth keys to the prow-job-analysis eval annotations
(expected_root_cause, expected_evidence[{string,artifact}], discriminating_signal,
distractors, must_not_conclude, plus expected_child_breakdown for aggregated jobs),
mirroring the payload-analysis eval's richness. Every expected_evidence string was
fetched from the public test-platform-results bucket and grep-confirmed verbatim.

- eval-prow-job-analysis.yaml: document all new annotation keys in dataset.schema;
  strengthen analysis_quality to require reaching expected_root_cause, citing
  expected_evidence, avoiding distractors, and penalizing must_not_conclude; add a
  new evidence_grounding LLM judge (min_mean 3.5) scoring grounded vs. hallucinated
  evidence. output_file_exists, required_skill_invocations, and
  failure_classification_accuracy are unchanged.
- case-001 (metal EPEL): verified EPEL mirror outage; corrected the make error to
  the verbatim "make: *** [Makefile:68: requirements] Error 2".
- case-002 (aggregator): corrected root cause — the failure is deterministic
  [Feature:NetworkSegmentation] UserDefinedNetwork conformance tests ("Passed 0
  times, failed 9 times"), NOT a disruption-threshold breach (disruption stayed
  within thresholds). References/category updated accordingly.

make lint: A+ (0 errors, 0 warnings).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ci): enrich prow-job-analysis eval cases 003-004 with verified ground truth

Apply the enriched annotation schema to case-003 and case-004, adding
expected_root_cause, expected_evidence[{string,artifact}], discriminating_signal,
distractors, and must_not_conclude. Every expected_evidence string was fetched
from the public test-platform-results bucket and grep-confirmed verbatim.

- case-003 (two-node-arbiter upgrade): corrected the root cause. The blocking
  upgrade invariant "ClusterOperators are available and not degraded after
  upgrade" failed because clusteroperator/network stayed Progressing ~11m47s
  with the openshift-multus/network-metrics-daemon and iptables-alerter
  DaemonSets "not available (awaiting 1 nodes)" -- NOT etcd connection storms
  (no such evidence). Switched secondary reference disruption.md -> networking.md
  and category to upgrade-network-operator-not-settled; disruption samples are
  documented as distractors.
- case-004 (hypershift release image import): confirmed the existing root cause
  and grounded it -- [release:initial] failed to reimport
  stable-initial:baremetal-installer because the backing digest was "not found"
  and the import timed out (reason importing_release). type/references/category
  unchanged.

make lint: A+ (0 errors, 0 warnings).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(ci): enrich + verify prow-job-analysis case-007/008 annotations

Rewrite case-007-cloud-quota-gcp-networks and
case-008-disconnected-payload-import to the verified annotation schema
(expected_root_cause, expected_evidence, discriminating_signal,
distractors, must_not_conclude). Every evidence string was fetched from
the public test-platform-results bucket and grep-confirmed verbatim.

case-007: confirmed root cause is a GCP NETWORKS quota exhaustion
(limit 50) hit by `gcloud compute networks create` in the
ipi-conf-gcp-xpn PRE step; classification unchanged (infra /
cloud-provider-errors.md / cloud-quota-exhaustion), notes tightened.

case-008: corrected a hallucinated root cause. The job did NOT fail in
the disconnected mirror path; ci-operator failed importing the
4.19.0-0.ci payload because member tag stable:baremetal-installer
(digest sha256:c4bc76ee...) was "not found" in the CI registry, timing
out after 6 imports before any test ran. Reclassified references
networking.md,hypershift.md -> hypershift.md,ci-infrastructure-changes.md
and category disconnected-payload-import -> release-image-import-failure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ci): enrich prow-job-analysis case-005/006 with verified schema

Re-verified both cases against real test-platform-results artifacts and
rewrote annotations.yaml to the enriched schema (expected_root_cause,
expected_evidence, discriminating_signal, distractors, must_not_conclude).
Every expected_evidence string was grep-confirmed verbatim in its cited GCS
artifact.

case-005 (ci-build-cluster-exhaustion): the openshift-e2e-test pod never
scheduled on the CI build farm (build09) and ci-operator failed with reason
pod_pending. Corrected the build cluster name (build09, not build01).

case-006 (mco-machineconfig-timeout): the commatrix "Nftables" test created
MachineConfig 98-nftables-commatrix-master and timed out after 1200s waiting
for the master MachineConfigPool to start updating (cluster.go:57). Corrected
the framing: no upgrade ran (ClusterVersion still 4.18, Progressing=False) and
the master MCP ended Updated=True/Degraded=False on a freshly rendered config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(ci): verify + enrich prow-job-analysis case-009 & case-010 annotations

Re-derive true root causes from real test-platform-results artifacts and
add the verified eval schema keys (expected_root_cause, expected_evidence,
discriminating_signal, distractors, must_not_conclude).

case-009 (multiarch 4.17->4.18 AWS OVN upgrade): corrected the hallucinated
"massive API/etcd disruption / 147+ dial tcp per etcd member" note. The
upgrade completed; the sole failure is the admin-ack gate test
("[bz-Cluster Version Operator] Verify presence of admin ack gate blocks
upgrade until acknowledged") timing out on the ClusterVersion
Upgradeable=AdminAckRequired condition (adminack.go:76), surfaced as a
goroutine Ginkgo panic. All disruption backend tests passed, so disruption
is a distractor; dropped disruption.md from expected_references.

case-010 (4.19 GCP OVN serial): nodes-readiness pre-step timed out at 6
expected / 5 found because worker Machine worker-c-b2tjb went phase Failed
after its GCP instance was created then went missing ("Instance not found
on provider" / "can't find created instance"). Replaced the wrong
resource-exhaustion.md reference with cloud-provider-errors.md.

Every expected_evidence.string was fetched from the public bucket and
grep-confirmed verbatim in its cited artifact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Rebuild prow-job-analysis evals with real TRT-incident cases

Replace the 10 placeholder prow-job-analysis eval cases with 11 cases built from real historical OpenShift CI failures. Each case carries grep-confirmed verbatim evidence from the public test-platform-results GCS bucket plus full structured annotations (expected_failure_type, expected_references, expected_root_cause, expected_evidence, discriminating_signal, distractors, must_not_conclude, and expected_child_breakdown for aggregated jobs).

Cases 001-011 cover: CRI-O CNI-rollout aggregated monitor regression (TRT-2723), openshift-tests-extension PENDING panic (TRT-2704), missing oc-mirror extension binary (TRT-2674), aggregated router-metrics regression (TRT-2757), HyperShift snapshot-controller CRD race (TRT-2702), vSphere TechPreview MCO render mismatch (TRT-2701), CRI-O RPM node-NotReady regression (TRT-2646), kube-apiserver non-graceful shutdown on upgrade (OCPBUGS-75200), metal OFCIR DNS outage (TRT-2665), Insights external-gateway HTTP 500 (TRT-2662), and EgressIP SNAT regression (TRT-2497).

Also document the input.yaml job_name and description fields in eval-prow-job-analysis.yaml's dataset schema.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(ci): replace required_skill_invocations judge with expected_references_used

Swap the programmatic required_skill_invocations judge in
eval-prow-job-analysis.yaml for a new expected_references_used judge that
verifies the analysis actually loaded the specialized reference file(s) the
routing table should have led it to.

The check parses the Claude Code stdout stream for Read tool calls that opened
files under the skill's references/ directory, maps each read to its path
relative to references/ (preserving the install/ subdirectory prefix, e.g.
install/metal.md), and compares that set against annotations.expected_references,
reporting which expected references were found and which were missed. It reads
ground truth via outputs["annotations"], matching the established convention used
by the address-reviews and classify-review-comment evals in this repo. The
thresholds entry is updated to expected_references_used (min_pass_rate: 1.0).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* eval(prow-job-analysis): broaden reference detection + fractional scoring

Update the expected_references_used check so it credits a reference as
loaded regardless of *how* the skill opened it, and score partial
coverage instead of hard-failing on any miss.

Detection now covers:
- Read tool calls (input.file_path under references/) — unchanged.
- Bash tool calls that read a reference via cat/tail/head/less/more/
  sed/awk/grep, excluding shallow `head -n N` peeks (N <= 20) that only
  skim a few lines.
- tool_result output that cites a reference path (targeted cat/grep/sed
  dump), while ignoring broad index/listing reads (e.g. the SKILL.md
  routing table) that name many references at once and would otherwise
  mark every reference "loaded".

Scoring is now fractional: pass when found/expected >= 0.5 so missing a
secondary reference no longer fails the case; the rationale always
reports the fraction, the references found, and those missed. The
expected_references_used threshold is relaxed from min_pass_rate 1.0 to
0.8 accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* evals(prow-job-analysis): trim expected_references to truly essential refs

Reduce each case's expected_references to only the deep-dive reference(s)
that own that case's root cause (1-2 per case), matching the intent of the
expected_references_used scorer (found/expected >= 0.5, min_pass_rate 0.8).

Drop generic waypoints (flaky-test-identification.md, which is explicitly a
router — "the deep-dive references own each root-cause class") except where
triage/classification is the case's core challenge (010), and drop
contextual references the annotations themselves flag as distractors.

Per case:
- 001: aggregated + operating-system-changes (drop upgrade; upgrade-mechanics is a distractor)
- 002: test-extension-binaries (drop flaky-test-identification)
- 003: test-extension-binaries (drop flaky-test-identification)
- 004: aggregated + networking (drop upgrade; upgrade-mechanics is a distractor)
- 005: hypershift (drop flaky-test-identification)
- 006: install/general + operating-system-changes (drop ci-infrastructure-changes)
- 007: operating-system-changes + upgrade (drop networking; CNI symptom is a distractor)
- 008: upgrade + operating-system-changes (drop disruption; flagged secondary)
- 009: install/metal + ci-infrastructure-changes (drop cloud-provider-errors; DNS outage != cloud quota/capacity)
- 010: unchanged (cloud-provider-errors + flaky-test-identification)
- 011: networking + test-failure (drop flaky-test-identification)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(prow-job-analysis): add OS-layer evidence check to core Step 5 triage

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: actionable SKILL description + correct OS-layer framing

* fix(ci): verify prow-job-analysis eval ground truth against artifacts; judge on facts, not verbatim strings

Fanned out one verification agent per eval case to re-derive every
annotation from the actual job artifacts in test-platform-results.
Findings and fixes:

Eval format (eval-prow-job-analysis.yaml):
- Replace expected_evidence {string, artifact} verbatim matching with
  key_evidence {fact, artifacts, example}: judges score whether a report
  ESTABLISHES each fact (equivalent lines, decoded/escaped variants, and
  faithful paraphrases all count) instead of playing spot-the-exact-string.
- Add acceptable_failure_types for genuine taxonomy-boundary cases and
  acceptable_references that are never penalized in either direction.
- Add optional depth_evidence (bonus-only deeper mechanism) and
  external_context (provenance from outside the artifacts; never a
  scoring requirement).
- Fabrication now means contradicting ground truth, not exceeding it.

Case corrections (all grep-verified against live artifacts):
- case-001: the required CRI-O 1.35->1.36 conclusion was CONTRADICTED by
  artifacts (cri-o 1.35.2-11 at install and end of run in child runs).
  Rewrote the expected root cause to the journal-provable mechanism:
  00-multus.conf REMOVE -> NotReady (NetworkPluginNotReady) -> CREATE
  within seconds during ovnkube/multus rollout.
- case-004: networking.md was mis-required (routing table says
  test-failure.md for a confirmed child-run regression); dropped three
  aggregator-boilerplate evidence strings; TRT-2757/#1439 moved to
  external_context (not in artifacts).
- case-006: "1797 additions" was a BSD-grep artifact — the real diff is
  3593 '+' lines / 0 '-'; VSphereMultiVCenterDay2 demoted from required
  evidence (inventory-dump-only, annotation both required and forbade it).
- case-007: annotations were CORRECT — the cri-o 1.35.2-11 -> 1.35.2-9
  version flip is fully provable in node journals; added the -9 side as
  evidence and the snapshot-hides-the-flip distractor.
- case-008: the systemd 90s stop-timeout SIGKILL chain IS in the master
  journals (annotation understated its own discoverability); added it as
  depth_evidence; upgrade|test both accepted.
- case-009: "shared-ci-dns-outage" loosened to ofcir-dns-resolution-failure
  (domain-wide outage not provable from one hostname); evidence citable
  from step log or top-level build-log.txt.
- case-010: cloud-provider-errors.md replaced with
  ci-infrastructure-changes.md per the routing table's own scoping;
  encoding-fragile evidence strings replaced with invariant cores.
- case-011: added the config.go:2460 'EnableEgressIP:false' startup dump
  as a distractor — it appears identically in PASSING runs.
- Scrubbed notes everywhere: judges render the whole annotations file, so
  notes no longer imply references or conclusions beyond the structured
  fields.

Skill fixes (root cause of the worst eval scores):
- SKILL.md OS-layer check + operating-system-changes.md + artifacts.md:
  node journals live at gather-extra/artifacts/nodes/<node>/journal and
  are gzip WITHOUT a .gz extension (zcat/zgrep, never plain grep);
  compare "Starting CRI-O, version" across boots within the run —
  end-of-run snapshots show only the final version and hide mid-run
  flips. journal_logs/ (the previously documented path) does not exist
  in these jobs.
- install/general.md: add machine-config RequiredPoolsFailed /
  bootstrap-vs-in-cluster rendered-MC mismatch pattern (was entirely
  uncovered).

Also regenerate docs (fixes the lint failure on #597) and bump ci to
0.0.67.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* evals(prow-job-analysis): drop case-001 and case-009

case-001's originating trigger (CRI-O behavior change) is not provable
from the job's artifacts, and case-009 is an infra DNS outage whose
precise scope likewise isn't decidable from a single run — neither
makes a clean, fully-groundable eval case. 9 cases remain; numbering
gaps left as-is.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* evals(prow-job-analysis): add case-012 build-farm disk-pressure eviction

Fills the resource-exhaustion.md routing gap. Real run where the
ipi-install-hosted-loki step pod was rejected at kubelet admission
(0s, empty failed-containers) with 'The node had condition:
[DiskPressure]' on a build09 farm node — before any cluster existed.
Exercises the two-failure-domains rule: build-farm exhaustion vs
cluster-under-test exhaustion. All evidence grep-verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* evals(prow-job-analysis): add case-013 ROSA NAT-gateway quota rejection

Fills the cloud-provider-errors.md routing gap and adds the suite's
first ROSA/OCM job. OCM rejects rosa create cluster with HTTP 400
CLUSTERS-MGMT-400 (NAT Gateways quota missing in us-east-2a) 12s into
the request; nothing provisioned; retry wrapper correctly classifies
it non-retryable. Distractors include a non-fatal jq error directly
above the failure and a synthesized junit_install.xml. All evidence
grep-verified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(prow-job-analysis): minimal runner prompt; mandatory OS-layer journal check

- Slim the runner system_prompt to the basics: use the skill, constrain
  code/config correlation to the job's start time (these are historical
  runs), write the report file. We are evaluating the skill, not a
  prompt — all coaching removed.
- Revert the accidental eval-payload-analysis.yaml dataset-path change
  (slipped in via git add -A during the case removals).
- SKILL.md: restructure the OS-layer evidence check into two mandatory
  numbered actions, the first being download-the-node-journals + the
  version-per-boot zgrep. The eval run on this PR showed the analysis
  never downloaded journals and cleared the OS layer from end-of-run
  snapshots — exactly the case-007 trap. Never clear the OS layer from
  snapshots alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* eval(prow-job-analysis): raise LLM-judge pass thresholds to 4.0

The repaired suite scores 4.18/4.36/4.36; 3.5 no longer guards anything.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(prow-job-analysis): download node journals in Step 4, not mid-analysis

Three eval runs show agents execute Step 4's download block verbatim
but skip the OS-layer check's own download even when marked mandatory
(case-007: 42 tool calls, zero journal access). Move journal
acquisition into the mechanical Step 4 block (|| true for jobs with no
cluster); Step 5a now only zgreps the already-local files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(prow-job-analysis): upgrade.md — check RPM changes across the upgrade boundary

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(prow-job-analysis): sweep references for taught-test content

Full sweep of SKILL.md + all 14 references for content that encodes
answers instead of teaching method:

- Unfounded frequency/ranking priors removed ('the #1 upgrade problem',
  'single most common cause/root cause', 'almost always/never' applied
  to root-cause likelihood) — rewritten as neutral mechanism
  descriptions with their existing confirmation steps intact.
- Single-string-to-verdict shortcuts now name the implicated layer and
  the confirming evidence instead of handing over a conclusion
  ('suspect X — confirm via Y').
- Eval-case answer leakage generalized to failure classes: the CRI-O
  version-flip narrative, 'Node became not ready'/'Stopping timed out'
  example greps, and the one-sided-MC-diff -> feature-gate-skew verdict
  are now class-level guidance (per-boot versions are recorded in
  journals; snapshots capture only final state; read the diff and trace
  the divergent rendering input).
- Kept: artifact structure, commands, deterministic facts (OOMKilled
  semantics, lease-before-product-code scoping, admission-rejection
  meaning), evidence-conditioned inferences, and the two-failure-
  domains rule.
- Fixed all anchors for the renamed 'MCO Drain Failures' heading.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(payload-analysis eval): case-007 blamed the wrong PR — it's OLM #1256, not CVO #1309

Independently verified against both failing metal job logs for payload
4.22.0-0.nightly-2026-03-18-161724: 26 extension binaries start listing,
25 complete — olmv0-tests-ext is the only one that never completes, so
the 'invalid character I' JSON corruption is its stdout.
cluster-version-operator-tests completes cleanly in BOTH jobs, which
exonerates the previously-annotated CVO#1309. operator-framework-olm#1256
(merged 2026-03-17, in payload) added e2e.Logf calls in the
olmv0-tests-ext client-setup path — klog INFO lines to stdout during OTE
listing. The skill has been answering this case correctly and getting
scored 1/5 for it. Renamed case dir to case-007-4.22-olm-revert.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(prow-job-analysis): gcloud is recommended, not required

Matches the setup script's optional handling and the bundled helper's
stdlib HTTP fallback. gcloud stays the documented fast path — eval
transcripts show all 25 cases across both suites used it — but the
public bucket needs no auth and everything works over plain HTTPS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(prow-job-analysis): metal archive names are .tar, not .tar.gz; soften case-003 notes

Verified against live metal jobs (three runs incl. the latest ipv4/ipv6
builds): CI gunzips artifacts, so libvirt-logs.tar, log-bundle-*.tar and
squid-logs-*.tar carry no .gz suffix (only sosreport keeps .tar.xz).
metal.md's .tar.gz references were the inconsistent ones —
operating-system-changes.md and general.md were already right. Also
mirror general.md's exploded log-bundle-*/ note in metal.md's Ironic
section, and drop the imperative 'NOT test-failure.md' clause from
case-003 notes (judges must never penalize extra reference reads).

Addresses review threads from coderabbitai and cblecker on #604.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(prow-job-analysis): address review feedback (python floor, metal sno scope, probe error surfacing)

- SKILL.md / continue-session.md: correct the Python floor to 3.7+ —
  prow_job_artifact_search.py uses add_subparsers(required=True), added in 3.7.
- metal.md: restrict the sno/single-node trigger to jobs that also carry a
  metal keyword so cloud SNO jobs (e.g. e2e-aws-ovn-single-node) don't route
  here; note the SKILL.md routing table is authoritative on conflict.
- setup-prow-job-analysis.sh: surface the GCS probe's actual error in the WARN
  instead of discarding it, to aid debugging in restricted-network environments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(prow-job-analysis eval): soften imperative notes in case-002 and case-004

The `notes` field is context-only per the eval schema, so it should describe
the scenario rather than issue MUST/MUST NOT directives to the agent (same fix
already applied to case-003 in b21a3f5). Rewrite case-002's "agent MUST
recognize … MUST NOT blame" and case-004's "agent MUST route to aggregated.md"
into descriptive statements of fact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Chai Bot <ship-help-github@redhat.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants