chore: sets the initial config for a market place and a hello world plugin - #1
Conversation
513a3ba to
2516425
Compare
| description: Hello world plugin implementation | ||
| --- | ||
|
|
||
| # Hello Command |
There was a problem hiding this comment.
In relation to this comment, what do you think about the this command being the definition for what new commands should look like?
I understand the purpose is to seed the marketplace but we could accomplish two goals with one PR.
There was a problem hiding this comment.
I really like this idea. I updated it to serve as a ref implementation
…lugin This enable consumers to install reusable pieces from this repo https://www.anthropic.com/news/claude-code-plugins https://anthropic.mintlify.app/en/docs/claude-code/plugin-marketplaces#how-to-configure-team-marketplaces https://anthropic.mintlify.app/en/docs/claude-code/plugins https://claudecodemarketplace.com/marketplace/anthropic-claude-code
2516425 to
3a7117a
Compare
|
/label lgtm |
|
@stbenjam: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
ptal @brandisher |
|
/lgtm |
|
@bryan-cox: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/lgtm |
|
@brandisher: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/lgtm |
|
@stbenjam: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/lgtm |
|
@stbenjam: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/lgtm |
|
@stbenjam: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/lgtm |
|
@stbenjam: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/lgtm |
|
@stbenjam: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Follow the section structure from PR openshift-eng#1: - ## Name - ## Synopsis - ## Description - ## Implementation - ## Return Value - ## Examples - ## Arguments This addresses review feedback to adhere to the established documentation format.
|
/lgtm |
|
@stbenjam: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/approve |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: brandisher, bryan-cox, enxebre, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Let AI to apply changes manually for conflicted files. Example: in the repo https://github.com/openshift/csi-operator.git , there is a commit (`9fa7ded32d45e71eb6715855ffc7f34852f375de`) in the branch `release-4.17`. The command `patch` failed in the branch `release-4.16`: ``` $ git show 9fa7ded32d45e71eb6715855ffc7f34852f375de | patch -p1 --no-backup-if-mismatch ... patching file assets/overlays/azure-disk/generated/hypershift/manifests.yaml Hunk openshift-eng#1 FAILED at 13. ``` However, Claude is smart enough to resolve the conflict manually. The result of imporved cherry-pick-by-patch is here: openshift/csi-operator@b236f6c
Let AI to apply changes manually for conflicted files. Example: in the repo https://github.com/openshift/csi-operator.git , there is a commit (`9fa7ded32d45e71eb6715855ffc7f34852f375de`) in the branch `release-4.17`. The command `patch` failed in the branch `release-4.16`: ``` $ git show 9fa7ded32d45e71eb6715855ffc7f34852f375de | patch -p1 --no-backup-if-mismatch ... patching file assets/overlays/azure-disk/generated/hypershift/manifests.yaml Hunk openshift-eng#1 FAILED at 13. ``` However, AI assistant is smart enough to resolve the conflict manually. The result of imporved cherry-pick-by-patch is here: openshift/csi-operator@b236f6c
Let AI to apply changes manually for conflicted files. Example: in the repo https://github.com/openshift/csi-operator.git , there is a commit (`9fa7ded32d45e71eb6715855ffc7f34852f375de`) in the branch `release-4.17`. The command `patch` failed in the branch `release-4.16`: ``` $ git show 9fa7ded32d45e71eb6715855ffc7f34852f375de | patch -p1 --no-backup-if-mismatch ... patching file assets/overlays/azure-disk/generated/hypershift/manifests.yaml Hunk openshift-eng#1 FAILED at 13. ``` However, AI assistant is smart enough to resolve the conflict manually. The result of imporved cherry-pick-by-patch is here: openshift/csi-operator@b236f6c
Blind test openshift-eng#1 found two critical issues: 1. Agent missed 6 test files when adding AtomicFIFO gate disabling because it only found SetFromMap-based files, not os.Setenv/t.Setenv ones. Add explicit grep command to find ALL files with existing gate disabling before adding new gates. 2. Agent migrated x/exp imports to stdlib but didn't run go mod tidy to remove x/exp from go.mod/vendor. Add explicit container-based cleanup instruction since local Go may be too old. Also fix t.Fatalf in init() example (t not available in init). Signed-off-by: Daniel Farrell <dfarrell@redhat.com>
… commands Addresses review feedback on the prow-job-analysis megaskill PR. BLOCKING openshift-eng#1 — dangling skill references: - Delete thin-wrapper commands analyze-prow-job-test-failure, analyze-prow-job-install-failure, extract-prow-job-must-gather (superseded by the prow-job-analysis skill; per review) - Repoint every reference to the 5 removed skills (prow-job-analyze-{test,install,metal-install}-failure, prow-job-artifact-search, prow-job-extract-must-gather) to ci:prow-job-analysis across commands, skills, references, README, PLUGINS.md, and the openshift/teams plugins - Fix the payload-analysis quality gate to match the single-skill model - Point analyze-disruption at the relocated prow_job_artifact_search.py BLOCKING openshift-eng#2 — must-gather ghost references: - Remove the no-op --fast flag (must-gather is no longer auto-extracted) - Correct the "renamed to content/ by extraction script" claims: the hash directory is preserved after tar -xf, not renamed - Mark the must-gather-analyzer scripts as run manually from the must-gather plugin (networking, resource-exhaustion, hypershift) - Add a "Must-Gather Download & Extraction" section (fetch tar, tar -xf, decompress nested archives) to the artifacts reference Bump ci 0.0.56->0.0.57 and openshift 0.0.6->0.0.7; sync marketplace.json and regenerate docs. make lint passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (supersedes #597) (#604) * feat(ci): add prow-job-analysis megaskill, remove 5 superseded skills Add the prow-job-analysis megaskill (SKILL.md plus reference guides for artifacts, aggregated jobs, CI-infrastructure changes, cloud-provider errors, disruption, flaky tests, HyperShift, install general/metal, networking, resource exhaustion, test-extension binaries, and upgrades). Remove the five skills it supersedes and relocate the artifact-search helper script into the megaskill: - prow-job-analyze-test-failure - prow-job-artifact-search (prow_job_artifact_search.py moved into the megaskill) - prow-job-analyze-install-failure - prow-job-analyze-metal-install-failure - prow-job-extract-must-gather Resolve CodeRabbit review feedback: - Tag every fenced code block with a language (MD040) - artifacts.md / ci-infrastructure-changes.md: {JOB_NAME} is the top-level bucket path only; step artifacts stay under artifacts/{target}/ - SKILL.md: align URL examples with the artifact-search parser contract (logs/) - networking.md: escape the pipe in the ICSP/IDMS table cell - upgrade.md: route bare-metal install failures to install/metal.md - hypershift.md: route Go-test/CI-step failures to build-log/step-graph analysis instead of JUnit XML / the extension-binary guide - aggregated.md: fix the contradictory "all children passed" failure mode - cloud-provider-errors.md: fix the Region/Zone Availability anchor links Bump the ci plugin to 0.0.56 and regenerate marketplace.json + docs/index.html. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): correct accuracy/cross-ref issues in prow-job-analysis review Self-review of the prow-job-analysis megaskill. Fixes verified against upstream sources (dev-scripts Makefile, baremetalds gather step, openshift-tests-extension types.go) and repo-internal grep. Accuracy (verified against source): - metal.md: swap dev-scripts script numbers — 03=build_installer, 04=setup_ironic (were reversed) across table, failure patterns, artifact map, quick-ref - metal.md: metal gather archives are gzipped — libvirt-logs.tar.gz, log-bundle-*.tar.gz, squid-logs-*.tar.gz; fix $-anchored greps that missed them; add installer-status.txt; add SNO bootstrap-in-place note; replace invented OFCIR build-log JSON/pool schema with grep guidance - test-extension-binaries.md: correct OTE JSON — result/passed (not status/pass), duration int64 ns, component type, snake_case source fields, apiVersion v1.1; replace rot-prone 34-row registry catalog with naming pattern + discovery command + verified exceptions - ci-infrastructure-changes.md / cloud-provider-errors.md: fix Boskos "(legacy)" and OFCIR mislabelling; OFCIR = OpenShift Fleeting CI Resources, consistent across files; Boskos brokers cloud quota slices - general.md: log bundles are gzipped (robust .tar(.gz)? matching) - upgrade.md: remove fabricated `drain exceeded timeout`/`5400s` MCD strings and false 90-min default; fix jq-on-YAML must-gather bugs (use yq / oc_cmds text); add EUS-to-EUS paused-MCP note - resource-exhaustion.md: fix gather-extra/oc_cmds path (missing /artifacts/); SystemOOM (not OOMKilling) for kubelet events - disruption.md: fix self-inconsistent interval JSON example Cross-references (dead-ends found via grep): - flaky-test-identification.md: drop dead importing_release→networking route; add HyperShift onward route; add utilizing_cluster_claim - hypershift.md: konnectivity is HCP-specific (networking.md lacks it); correct See Also descriptions to what siblings actually cover - networking.md: retarget OVS-stall ref to disruption.md; move misfiled Hairpinning bullet from DNS to Service Connectivity Progressive disclosure / conciseness: - general.md: drop duplicated per-mode procedure block - metal.md: point to artifacts.md for canonical metal layout - SKILL.md: add mkdir -p so download commands work make lint: 0 errors, 0 warnings, Grade A+, all checks passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): repoint dangling refs to removed skills, delete thin-wrapper commands Addresses review feedback on the prow-job-analysis megaskill PR. BLOCKING #1 — dangling skill references: - Delete thin-wrapper commands analyze-prow-job-test-failure, analyze-prow-job-install-failure, extract-prow-job-must-gather (superseded by the prow-job-analysis skill; per review) - Repoint every reference to the 5 removed skills (prow-job-analyze-{test,install,metal-install}-failure, prow-job-artifact-search, prow-job-extract-must-gather) to ci:prow-job-analysis across commands, skills, references, README, PLUGINS.md, and the openshift/teams plugins - Fix the payload-analysis quality gate to match the single-skill model - Point analyze-disruption at the relocated prow_job_artifact_search.py BLOCKING #2 — must-gather ghost references: - Remove the no-op --fast flag (must-gather is no longer auto-extracted) - Correct the "renamed to content/ by extraction script" claims: the hash directory is preserved after tar -xf, not renamed - Mark the must-gather-analyzer scripts as run manually from the must-gather plugin (networking, resource-exhaustion, hypershift) - Add a "Must-Gather Download & Extraction" section (fetch tar, tar -xf, decompress nested archives) to the artifacts reference Bump ci 0.0.56->0.0.57 and openshift 0.0.6->0.0.7; sync marketplace.json and regenerate docs. make lint passes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ci): apply terseness cuts to prow-job-analysis references Remove verified padding (restatement, generic reasoning, redundant checklists) from five reference files while preserving all domain strings, error messages, tables, grep patterns, and Quick References. - ci-infrastructure-changes.md: add a Table of Contents; collapse the third-pass "CI Job Lifecycle" phase breakdown to the key rule; drop the "Common CI Infrastructure Failure Patterns" re-describe (folding the new Step Script Bug guidance into the earlier indicators); cut the closing classification checklist. - install/metal.md: delete "Tips and Best Practices" (all restatements); collapse grep cookbooks to keyword lists; keep one download recipe. - upgrade.md: replace "Putting It All Together" with a compact ordered list pointing at the Key Files table (grep markers + parser command preserved); trim the 4x-repeated ClusterOperator JSON to its humanMessage patterns. - aggregated.md: keep one parent-build-log warning; cut the rationale essay and self-evident glossary; compress "Multi-Step vs True Aggregated" (defer step-registry to ci-infrastructure-changes.md); drop the duplicate checklist. - disruption.md: compress the disruption-causation checklist; drop Scenarios A/B/C (folding C's nugget into the upgrade-phase list); cut "Why Cross-Run Analysis Matters". Cross-file dedup: - parse_disruption.py docs now link to analyze-disruption/SKILL.md. - artifacts.md interval format links to disruption.md (path row kept). - job_labels download commands owned by artifacts.md; others link to it. Net -569 lines. Bump ci 0.0.57->0.0.58; sync marketplace.json and regenerate docs. make lint passes (0 errors, 0 warnings, A+). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ci): add test-failure root-cause reference; close routing dead-end flaky-test-identification.md is the "start here" for any failing test, but once it classified a failure as a real product regression in a plain (non-extension, non-install, non-upgrade) e2e test, its onward routes went nowhere — the deleted prow-job-analyze-test-failure skill's core methodology had no home. Add references/test-failure.md owning plain e2e regression root-cause: - locate test source from the Go/Ginkgo test name (openshift/origin test/extended, vendored k8s conformance, extension codeLocations) - read JUnit failure message + stack-trace shape - scope namespace/component from the [sig-*] prefix - pin the failure window from interval files and temporally correlate cluster/operator events - trace crash-looping containers via pod containerStatuses (exitCode 137/143/1/2, reason), previous.log, dependency chain - "never stop at high-level symptoms" synthesis - route to networking/resource-exhaustion/disruption/cloud references when the root cause lands in another domain Wire it into the routing tables: - SKILL.md failure routing table - flaky-test-identification.md: product-regression class, "use a different reference" line, JUnit product-behavior sentence, triage-checklist hand-off, and See Also Bump ci 0.0.58 -> 0.0.59; sync marketplace.json and regenerate docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ci): add operating-system-changes (RHCOS) reference to prow-job-analysis Add references/operating-system-changes.md covering RHCOS-related failure debugging, as requested in PR review: when and why to start debugging OS-level issues (odd cri-o/container-start behavior, NetworkManager networking problems, kernel panics, systemd/SELinux, storage/rpm-ostree). The reference explains RHCOS as the immutable, rpm-ostree-managed OS shipped as a payload component (rhel-coreos / rhel-coreos-10), how an OS change propagates via the MCO/MCD (drain -> deploy -> reboot), and the highest-signal ways to suspect the OS layer: RHCOS variant isolation (RHCOS 9 vs 10) and payload-boundary blast radius. For payloads it points at the deep RPM-level changelog available in the snapshot (release controller nodeImageStreams, summary.json rhcos_changes[], rpmdb.sqlite), and notes RHCOS RPM changes are suspects for the platform team, not revert candidates. Includes failure-pattern and artifact-location tables (journals, host_service_logs, serial console, failed-units, MCO logs) and a quick triage checklist. Deep MCO drain mechanics stay owned by upgrade.md to avoid duplication. Wire it into SKILL.md: a Failure Routing Table row and a job-name classification row (rhcos9/rhcos10/rhcos9_10/rt). Bump ci 0.0.59->0.0.60; sync marketplace.json and regenerate docs. make lint passes (0 errors, 0 warnings, A+). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ci): unify payload-analysis subagent routing to prow-job-analysis The payload-analysis subagent prompt split failed-job investigation into "install failure" vs "test failure" branches, each pointing at `ci:prow-job-analysis` separately, and pre-classified the failure via JUnit before choosing. That routing is redundant: prow-job-analysis already identifies the job type, classifies the failure, and routes to the specialized reference (install, metal, test, upgrade, etc.) internally. - Replace the install/test bifurcation with one imperative directive to use `ci:prow-job-analysis` for every failed job. - Drop the "First, check the JUnit results..." pre-classification step; that is prow-job-analysis's job now. - Keep the trace-to-root-cause and infrastructure-flake guidance (analysis quality, not routing). Also delete docs/data.json, a stale orphaned artifact removed during the skillsaw migration (9dd2118) but later resurrected. Nothing regenerates it (skillsaw docs writes only index.html) and nothing references it (index.html fetches live data). It still listed the four deleted prow-job-analyze-*/artifact-search/extract-must-gather skills -- the only remaining dangling references to them in the repo. Bump ci plugin 0.0.60 -> 0.0.61 and sync marketplace/docs via `make update`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address stbenjam & coderabbit review on prow-job-analysis references Reference-file corrections from PR #597 review: - operating-system-changes.md: reword "Use when" opener as a diagnosis of exclusion (you don't know a failure is OS-level until other modes are ruled out) instead of asserting the origin up front. - install/general.md & install/metal.md: make the imperative "Use when..." the first prose line under the title. - install/general.md: "also see metal.md" -> "you must read metal.md"; add major-upgrade (4->5) case; clarify {bucket-path} is the specific job run's path (going up a level lists tens of thousands of jobs); note log bundles are stored decompressed (never .tar.gz) and are now uploaded exploded as a directory; defer must-gather detail to the artifacts reference. - artifacts.md: aggregated JUnit path uses {underlying-job-name}. - test-extension-binaries.md: step-log path uses {target}, not {JOB_NAME}. Bump ci plugin 0.0.61 -> 0.0.62 and regenerate docs/marketplace. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): incorporate not-stbenjam eval review into payload-analysis Applies 8 generalized behavioral improvements to the payload-analysis SKILL.md to reduce wrong-PR attribution and confidence inflation: 1. Conflict-adjudication protocol (Step 5b) — contradictory root causes for the same failure signature are UNRESOLVED, not vibe-broken ties; require discriminating evidence from the exact failing operation, "cleared" needs positive evidence from the failing code path, absent log lines aren't evidence under truncation, causal chains must be shown to execute, and overrides update the stored per-job root cause. 2. Rubric de-duplication and tiering (Step 6.1) — fold the redundant "single candidate" signal into component exclusivity; tier error-message match (+40 direct / +20-30 same code path / +10 same subsystem); require an itemized per-signal breakdown; re-verify top-tier claims when the raw sum exceeds the cap. 3. Tighten force-accept (Step 6.4) — moot when phase is already Accepted; define "temporary" via self-resolve-without-human-action. 4. Score every distinct failure mode, not just the dominant one; "intermittent/flake" is a conclusion requiring evidence. 5. Causal CI-infrastructure changes must appear as scored candidates[] entries even when failure_type is infra (Steps 3.6 and 6.5). 6. Infrastructure exclusion — unrelated PRs score at/near zero for affirmatively infra-caused failures. 7. Surface payload-chain context (consecutive rejections, last accepted payload, per-job persistence) in the executive summary (Step 7.1). 8. Final mechanical self-check and verbatim phase recording (Steps 3.1, Step 10). Bumps ci plugin 0.0.62 -> 0.0.63; syncs marketplace.json and docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): anchor payload-analysis output dir, tighten scoring; finish stbenjam reference fixes payload-analysis SKILL.md: - Step 1 captures OUTPUT_DIR="$(pwd)" before anything else; the results YAML (6.5), HTML summary (7), and autodl JSON (8) are all written under $OUTPUT_DIR. - Step 10 self-check verifies the three exact spec'd filenames at $OUTPUT_DIR (no glob) and first confirms all Step 4 subagents and the Step 9 reviewer completed before assembling the report. - Step 6.1: recorded confidence score MUST equal min(100, sum of itemized signals), each signal at its defined weight with one line of evidence; enumerate every failure mode per job before scoring and give each its own rubric pass; flake dismissals must cite evidence. - Step 6.1: when a job has zero causally-linked candidates, require a per-job rationale and an explicit cross-job correlation statement. prow-job-analysis references: - test-failure.md: drop the unhelpful emphasis asterisks and trim the verbose opener (stbenjam). - operating-system-changes.md: reword trigger to a diagnosis of exclusion ("Use when no other failure mode explains the cause..."). - upgrade.md: open with an imperative "Use when" trigger as the first prose line and fold the redundant "When to Use" list into it. Version bump ci 0.0.63 -> 0.0.64 via make update (marketplace.json + docs). make lint: A+, 0 errors, 0 warnings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): add prow-job-analysis eval with 10 failure-mode cases Adds a skillsaw eval for the ci:prow-job-analysis skill, modeled on eval-payload-analysis.yaml. - eval-prow-job-analysis.yaml: mode=case eval that passes each case's prow_job_url to ci:prow-job-analysis and has the runner write a prow-job-analysis-report.md. Judges: two deterministic (output_file_exists, required_skill_invocations) and two LLM rubrics (analysis_quality 1-5, failure_classification_accuracy 1-5), each with a threshold. Dataset points at cases/prow-job-analysis/. - scripts/setup-prow-job-analysis.sh: verifies gcloud/python3/jq, confirms the public test-platform-results bucket is reachable, prepares the working dir (no snapshot needed — artifacts are read live). - 10 case dirs, each with input.yaml (plausible Prow URL) and annotations.yaml (expected_failure_type, expected_references, expected_root_cause_category, notes): metal install/Ironic, aggregated mixed children, upgrade MCO drain, hypershift dual-cluster, CI step-registry infra change, extension-binary crash, cloud quota exhaustion, disconnected proxy pull, upgrade API disruption, and SNO OOM. Spread: install x3, upgrade x2, test x3, infra x2. Bump ci plugin 0.0.64 -> 0.0.65 and sync marketplace/docs. make lint: A+. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): make prow-job-analysis artifact access work without gcloud The eval-prow-job-analysis job failed because setup-prow-job-analysis.sh required the gcloud CLI (which is not installed in the CI eval container) and prow_job_artifact_search.py used `gcloud storage` for all GCS access. The test-platform-results bucket is public, so no auth or SDK is needed. Add a Python-stdlib (urllib) fallback to the public GCS JSON/download API, used automatically whenever gcloud is unavailable: - prow_job_artifact_search.py: list/search/fetch fall back to the public GCS HTTP API when gcloud is absent (or PROW_ARTIFACT_SEARCH_NO_GCLOUD=1). JSON output is identical to the gcloud path. Verified against gcloud on a real job (search **/*.json 227==227, **/*.tar 8==8, **/nodes 1==1, list matches the GCS API exactly); fetch reports the decompressed size for gzip-transcoded artifacts. - setup-prow-job-analysis.sh: gcloud is now optional (python3/jq required); bucket reachability and pre-warm use the public API / bundled script, so the script no longer exits 2 when gcloud is missing. - Bump ci plugin 0.0.65 -> 0.0.66; sync marketplace.json and regen docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ci): replace hallucinated prow-job-analysis eval cases with real jobs The 10 prow-job-analysis eval cases used fabricated sequential build IDs (1780146820004390401-410) that do not exist in GCS. Replace all of them with real, failed CI jobs whose artifacts are present in the public test-platform-results bucket (each verified: build-log.txt returns HTTP 200 and finished.json reports result=FAILURE). Cases now cover: metal IPI install (EPEL mirror outage), aggregated mixed children (disruption threshold), two-node-arbiter upgrade operator degradation, HyperShift release-image import failure, CI build-cluster scheduling exhaustion, MCO/MachineConfigPool timeout, GCP NETWORKS quota exhaustion, disconnected payload import, multi-arch upgrade API disruption, and GCP node-not-ready timeout. Each input.yaml carries a real prow_job_url and each annotations.yaml sets expected_failure_type, expected_references, expected_root_cause_category, and notes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(images): add gcloud CLI to container image Install the google-cloud-cli package in both images/Dockerfile and images/Dockerfile.nested-podman so gcloud (and the bundled gsutil and bq binaries) are available in the image. This is needed for accessing public GCS artifacts during Prow job analysis via `gcloud storage` and `gsutil`. Follows the existing external-repo pattern: adds images/repos/google-cloud-cli.repo plus a bundled GPG key at images/repos/RPM-GPG-KEY-google-cloud. The key file contains BOTH of Google's signing keys as separate armor blocks (like RPM-GPG-KEY-gh-cli): the RPM package-signing key (used by gpgcheck) and the Artifact Registry metadata-signing key (used by repo_gpgcheck), so both gpgcheck=1 and repo_gpgcheck=1 succeed. The new .repo and key files are picked up automatically by the existing COPY globs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): enrich prow-job-analysis eval schema + verified case-001/002 fixtures Add structured ground-truth keys to the prow-job-analysis eval annotations (expected_root_cause, expected_evidence[{string,artifact}], discriminating_signal, distractors, must_not_conclude, plus expected_child_breakdown for aggregated jobs), mirroring the payload-analysis eval's richness. Every expected_evidence string was fetched from the public test-platform-results bucket and grep-confirmed verbatim. - eval-prow-job-analysis.yaml: document all new annotation keys in dataset.schema; strengthen analysis_quality to require reaching expected_root_cause, citing expected_evidence, avoiding distractors, and penalizing must_not_conclude; add a new evidence_grounding LLM judge (min_mean 3.5) scoring grounded vs. hallucinated evidence. output_file_exists, required_skill_invocations, and failure_classification_accuracy are unchanged. - case-001 (metal EPEL): verified EPEL mirror outage; corrected the make error to the verbatim "make: *** [Makefile:68: requirements] Error 2". - case-002 (aggregator): corrected root cause — the failure is deterministic [Feature:NetworkSegmentation] UserDefinedNetwork conformance tests ("Passed 0 times, failed 9 times"), NOT a disruption-threshold breach (disruption stayed within thresholds). References/category updated accordingly. make lint: A+ (0 errors, 0 warnings). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): enrich prow-job-analysis eval cases 003-004 with verified ground truth Apply the enriched annotation schema to case-003 and case-004, adding expected_root_cause, expected_evidence[{string,artifact}], discriminating_signal, distractors, and must_not_conclude. Every expected_evidence string was fetched from the public test-platform-results bucket and grep-confirmed verbatim. - case-003 (two-node-arbiter upgrade): corrected the root cause. The blocking upgrade invariant "ClusterOperators are available and not degraded after upgrade" failed because clusteroperator/network stayed Progressing ~11m47s with the openshift-multus/network-metrics-daemon and iptables-alerter DaemonSets "not available (awaiting 1 nodes)" -- NOT etcd connection storms (no such evidence). Switched secondary reference disruption.md -> networking.md and category to upgrade-network-operator-not-settled; disruption samples are documented as distractors. - case-004 (hypershift release image import): confirmed the existing root cause and grounded it -- [release:initial] failed to reimport stable-initial:baremetal-installer because the backing digest was "not found" and the import timed out (reason importing_release). type/references/category unchanged. make lint: A+ (0 errors, 0 warnings). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): enrich + verify prow-job-analysis case-007/008 annotations Rewrite case-007-cloud-quota-gcp-networks and case-008-disconnected-payload-import to the verified annotation schema (expected_root_cause, expected_evidence, discriminating_signal, distractors, must_not_conclude). Every evidence string was fetched from the public test-platform-results bucket and grep-confirmed verbatim. case-007: confirmed root cause is a GCP NETWORKS quota exhaustion (limit 50) hit by `gcloud compute networks create` in the ipi-conf-gcp-xpn PRE step; classification unchanged (infra / cloud-provider-errors.md / cloud-quota-exhaustion), notes tightened. case-008: corrected a hallucinated root cause. The job did NOT fail in the disconnected mirror path; ci-operator failed importing the 4.19.0-0.ci payload because member tag stable:baremetal-installer (digest sha256:c4bc76ee...) was "not found" in the CI registry, timing out after 6 imports before any test ran. Reclassified references networking.md,hypershift.md -> hypershift.md,ci-infrastructure-changes.md and category disconnected-payload-import -> release-image-import-failure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): enrich prow-job-analysis case-005/006 with verified schema Re-verified both cases against real test-platform-results artifacts and rewrote annotations.yaml to the enriched schema (expected_root_cause, expected_evidence, discriminating_signal, distractors, must_not_conclude). Every expected_evidence string was grep-confirmed verbatim in its cited GCS artifact. case-005 (ci-build-cluster-exhaustion): the openshift-e2e-test pod never scheduled on the CI build farm (build09) and ci-operator failed with reason pod_pending. Corrected the build cluster name (build09, not build01). case-006 (mco-machineconfig-timeout): the commatrix "Nftables" test created MachineConfig 98-nftables-commatrix-master and timed out after 1200s waiting for the master MachineConfigPool to start updating (cluster.go:57). Corrected the framing: no upgrade ran (ClusterVersion still 4.18, Progressing=False) and the master MCP ended Updated=True/Degraded=False on a freshly rendered config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(ci): verify + enrich prow-job-analysis case-009 & case-010 annotations Re-derive true root causes from real test-platform-results artifacts and add the verified eval schema keys (expected_root_cause, expected_evidence, discriminating_signal, distractors, must_not_conclude). case-009 (multiarch 4.17->4.18 AWS OVN upgrade): corrected the hallucinated "massive API/etcd disruption / 147+ dial tcp per etcd member" note. The upgrade completed; the sole failure is the admin-ack gate test ("[bz-Cluster Version Operator] Verify presence of admin ack gate blocks upgrade until acknowledged") timing out on the ClusterVersion Upgradeable=AdminAckRequired condition (adminack.go:76), surfaced as a goroutine Ginkgo panic. All disruption backend tests passed, so disruption is a distractor; dropped disruption.md from expected_references. case-010 (4.19 GCP OVN serial): nodes-readiness pre-step timed out at 6 expected / 5 found because worker Machine worker-c-b2tjb went phase Failed after its GCP instance was created then went missing ("Instance not found on provider" / "can't find created instance"). Replaced the wrong resource-exhaustion.md reference with cloud-provider-errors.md. Every expected_evidence.string was fetched from the public bucket and grep-confirmed verbatim in its cited artifact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Rebuild prow-job-analysis evals with real TRT-incident cases Replace the 10 placeholder prow-job-analysis eval cases with 11 cases built from real historical OpenShift CI failures. Each case carries grep-confirmed verbatim evidence from the public test-platform-results GCS bucket plus full structured annotations (expected_failure_type, expected_references, expected_root_cause, expected_evidence, discriminating_signal, distractors, must_not_conclude, and expected_child_breakdown for aggregated jobs). Cases 001-011 cover: CRI-O CNI-rollout aggregated monitor regression (TRT-2723), openshift-tests-extension PENDING panic (TRT-2704), missing oc-mirror extension binary (TRT-2674), aggregated router-metrics regression (TRT-2757), HyperShift snapshot-controller CRD race (TRT-2702), vSphere TechPreview MCO render mismatch (TRT-2701), CRI-O RPM node-NotReady regression (TRT-2646), kube-apiserver non-graceful shutdown on upgrade (OCPBUGS-75200), metal OFCIR DNS outage (TRT-2665), Insights external-gateway HTTP 500 (TRT-2662), and EgressIP SNAT regression (TRT-2497). Also document the input.yaml job_name and description fields in eval-prow-job-analysis.yaml's dataset schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(ci): replace required_skill_invocations judge with expected_references_used Swap the programmatic required_skill_invocations judge in eval-prow-job-analysis.yaml for a new expected_references_used judge that verifies the analysis actually loaded the specialized reference file(s) the routing table should have led it to. The check parses the Claude Code stdout stream for Read tool calls that opened files under the skill's references/ directory, maps each read to its path relative to references/ (preserving the install/ subdirectory prefix, e.g. install/metal.md), and compares that set against annotations.expected_references, reporting which expected references were found and which were missed. It reads ground truth via outputs["annotations"], matching the established convention used by the address-reviews and classify-review-comment evals in this repo. The thresholds entry is updated to expected_references_used (min_pass_rate: 1.0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * eval(prow-job-analysis): broaden reference detection + fractional scoring Update the expected_references_used check so it credits a reference as loaded regardless of *how* the skill opened it, and score partial coverage instead of hard-failing on any miss. Detection now covers: - Read tool calls (input.file_path under references/) — unchanged. - Bash tool calls that read a reference via cat/tail/head/less/more/ sed/awk/grep, excluding shallow `head -n N` peeks (N <= 20) that only skim a few lines. - tool_result output that cites a reference path (targeted cat/grep/sed dump), while ignoring broad index/listing reads (e.g. the SKILL.md routing table) that name many references at once and would otherwise mark every reference "loaded". Scoring is now fractional: pass when found/expected >= 0.5 so missing a secondary reference no longer fails the case; the rationale always reports the fraction, the references found, and those missed. The expected_references_used threshold is relaxed from min_pass_rate 1.0 to 0.8 accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * evals(prow-job-analysis): trim expected_references to truly essential refs Reduce each case's expected_references to only the deep-dive reference(s) that own that case's root cause (1-2 per case), matching the intent of the expected_references_used scorer (found/expected >= 0.5, min_pass_rate 0.8). Drop generic waypoints (flaky-test-identification.md, which is explicitly a router — "the deep-dive references own each root-cause class") except where triage/classification is the case's core challenge (010), and drop contextual references the annotations themselves flag as distractors. Per case: - 001: aggregated + operating-system-changes (drop upgrade; upgrade-mechanics is a distractor) - 002: test-extension-binaries (drop flaky-test-identification) - 003: test-extension-binaries (drop flaky-test-identification) - 004: aggregated + networking (drop upgrade; upgrade-mechanics is a distractor) - 005: hypershift (drop flaky-test-identification) - 006: install/general + operating-system-changes (drop ci-infrastructure-changes) - 007: operating-system-changes + upgrade (drop networking; CNI symptom is a distractor) - 008: upgrade + operating-system-changes (drop disruption; flagged secondary) - 009: install/metal + ci-infrastructure-changes (drop cloud-provider-errors; DNS outage != cloud quota/capacity) - 010: unchanged (cloud-provider-errors + flaky-test-identification) - 011: networking + test-failure (drop flaky-test-identification) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(prow-job-analysis): add OS-layer evidence check to core Step 5 triage Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: actionable SKILL description + correct OS-layer framing * fix(ci): verify prow-job-analysis eval ground truth against artifacts; judge on facts, not verbatim strings Fanned out one verification agent per eval case to re-derive every annotation from the actual job artifacts in test-platform-results. Findings and fixes: Eval format (eval-prow-job-analysis.yaml): - Replace expected_evidence {string, artifact} verbatim matching with key_evidence {fact, artifacts, example}: judges score whether a report ESTABLISHES each fact (equivalent lines, decoded/escaped variants, and faithful paraphrases all count) instead of playing spot-the-exact-string. - Add acceptable_failure_types for genuine taxonomy-boundary cases and acceptable_references that are never penalized in either direction. - Add optional depth_evidence (bonus-only deeper mechanism) and external_context (provenance from outside the artifacts; never a scoring requirement). - Fabrication now means contradicting ground truth, not exceeding it. Case corrections (all grep-verified against live artifacts): - case-001: the required CRI-O 1.35->1.36 conclusion was CONTRADICTED by artifacts (cri-o 1.35.2-11 at install and end of run in child runs). Rewrote the expected root cause to the journal-provable mechanism: 00-multus.conf REMOVE -> NotReady (NetworkPluginNotReady) -> CREATE within seconds during ovnkube/multus rollout. - case-004: networking.md was mis-required (routing table says test-failure.md for a confirmed child-run regression); dropped three aggregator-boilerplate evidence strings; TRT-2757/#1439 moved to external_context (not in artifacts). - case-006: "1797 additions" was a BSD-grep artifact — the real diff is 3593 '+' lines / 0 '-'; VSphereMultiVCenterDay2 demoted from required evidence (inventory-dump-only, annotation both required and forbade it). - case-007: annotations were CORRECT — the cri-o 1.35.2-11 -> 1.35.2-9 version flip is fully provable in node journals; added the -9 side as evidence and the snapshot-hides-the-flip distractor. - case-008: the systemd 90s stop-timeout SIGKILL chain IS in the master journals (annotation understated its own discoverability); added it as depth_evidence; upgrade|test both accepted. - case-009: "shared-ci-dns-outage" loosened to ofcir-dns-resolution-failure (domain-wide outage not provable from one hostname); evidence citable from step log or top-level build-log.txt. - case-010: cloud-provider-errors.md replaced with ci-infrastructure-changes.md per the routing table's own scoping; encoding-fragile evidence strings replaced with invariant cores. - case-011: added the config.go:2460 'EnableEgressIP:false' startup dump as a distractor — it appears identically in PASSING runs. - Scrubbed notes everywhere: judges render the whole annotations file, so notes no longer imply references or conclusions beyond the structured fields. Skill fixes (root cause of the worst eval scores): - SKILL.md OS-layer check + operating-system-changes.md + artifacts.md: node journals live at gather-extra/artifacts/nodes/<node>/journal and are gzip WITHOUT a .gz extension (zcat/zgrep, never plain grep); compare "Starting CRI-O, version" across boots within the run — end-of-run snapshots show only the final version and hide mid-run flips. journal_logs/ (the previously documented path) does not exist in these jobs. - install/general.md: add machine-config RequiredPoolsFailed / bootstrap-vs-in-cluster rendered-MC mismatch pattern (was entirely uncovered). Also regenerate docs (fixes the lint failure on #597) and bump ci to 0.0.67. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * evals(prow-job-analysis): drop case-001 and case-009 case-001's originating trigger (CRI-O behavior change) is not provable from the job's artifacts, and case-009 is an infra DNS outage whose precise scope likewise isn't decidable from a single run — neither makes a clean, fully-groundable eval case. 9 cases remain; numbering gaps left as-is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * evals(prow-job-analysis): add case-012 build-farm disk-pressure eviction Fills the resource-exhaustion.md routing gap. Real run where the ipi-install-hosted-loki step pod was rejected at kubelet admission (0s, empty failed-containers) with 'The node had condition: [DiskPressure]' on a build09 farm node — before any cluster existed. Exercises the two-failure-domains rule: build-farm exhaustion vs cluster-under-test exhaustion. All evidence grep-verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * evals(prow-job-analysis): add case-013 ROSA NAT-gateway quota rejection Fills the cloud-provider-errors.md routing gap and adds the suite's first ROSA/OCM job. OCM rejects rosa create cluster with HTTP 400 CLUSTERS-MGMT-400 (NAT Gateways quota missing in us-east-2a) 12s into the request; nothing provisioned; retry wrapper correctly classifies it non-retryable. Distractors include a non-fatal jq error directly above the failure and a synthesized junit_install.xml. All evidence grep-verified. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(prow-job-analysis): minimal runner prompt; mandatory OS-layer journal check - Slim the runner system_prompt to the basics: use the skill, constrain code/config correlation to the job's start time (these are historical runs), write the report file. We are evaluating the skill, not a prompt — all coaching removed. - Revert the accidental eval-payload-analysis.yaml dataset-path change (slipped in via git add -A during the case removals). - SKILL.md: restructure the OS-layer evidence check into two mandatory numbered actions, the first being download-the-node-journals + the version-per-boot zgrep. The eval run on this PR showed the analysis never downloaded journals and cleared the OS layer from end-of-run snapshots — exactly the case-007 trap. Never clear the OS layer from snapshots alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * eval(prow-job-analysis): raise LLM-judge pass thresholds to 4.0 The repaired suite scores 4.18/4.36/4.36; 3.5 no longer guards anything. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prow-job-analysis): download node journals in Step 4, not mid-analysis Three eval runs show agents execute Step 4's download block verbatim but skip the OS-layer check's own download even when marked mandatory (case-007: 42 tool calls, zero journal access). Move journal acquisition into the mechanical Step 4 block (|| true for jobs with no cluster); Step 5a now only zgreps the already-local files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(prow-job-analysis): upgrade.md — check RPM changes across the upgrade boundary Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(prow-job-analysis): sweep references for taught-test content Full sweep of SKILL.md + all 14 references for content that encodes answers instead of teaching method: - Unfounded frequency/ranking priors removed ('the #1 upgrade problem', 'single most common cause/root cause', 'almost always/never' applied to root-cause likelihood) — rewritten as neutral mechanism descriptions with their existing confirmation steps intact. - Single-string-to-verdict shortcuts now name the implicated layer and the confirming evidence instead of handing over a conclusion ('suspect X — confirm via Y'). - Eval-case answer leakage generalized to failure classes: the CRI-O version-flip narrative, 'Node became not ready'/'Stopping timed out' example greps, and the one-sided-MC-diff -> feature-gate-skew verdict are now class-level guidance (per-boot versions are recorded in journals; snapshots capture only final state; read the diff and trace the divergent rendering input). - Kept: artifact structure, commands, deterministic facts (OOMKilled semantics, lease-before-product-code scoping, admission-rejection meaning), evidence-conditioned inferences, and the two-failure- domains rule. - Fixed all anchors for the renamed 'MCO Drain Failures' heading. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(payload-analysis eval): case-007 blamed the wrong PR — it's OLM #1256, not CVO #1309 Independently verified against both failing metal job logs for payload 4.22.0-0.nightly-2026-03-18-161724: 26 extension binaries start listing, 25 complete — olmv0-tests-ext is the only one that never completes, so the 'invalid character I' JSON corruption is its stdout. cluster-version-operator-tests completes cleanly in BOTH jobs, which exonerates the previously-annotated CVO#1309. operator-framework-olm#1256 (merged 2026-03-17, in payload) added e2e.Logf calls in the olmv0-tests-ext client-setup path — klog INFO lines to stdout during OTE listing. The skill has been answering this case correctly and getting scored 1/5 for it. Renamed case dir to case-007-4.22-olm-revert. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(prow-job-analysis): gcloud is recommended, not required Matches the setup script's optional handling and the bundled helper's stdlib HTTP fallback. gcloud stays the documented fast path — eval transcripts show all 25 cases across both suites used it — but the public bucket needs no auth and everything works over plain HTTPS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prow-job-analysis): metal archive names are .tar, not .tar.gz; soften case-003 notes Verified against live metal jobs (three runs incl. the latest ipv4/ipv6 builds): CI gunzips artifacts, so libvirt-logs.tar, log-bundle-*.tar and squid-logs-*.tar carry no .gz suffix (only sosreport keeps .tar.xz). metal.md's .tar.gz references were the inconsistent ones — operating-system-changes.md and general.md were already right. Also mirror general.md's exploded log-bundle-*/ note in metal.md's Ironic section, and drop the imperative 'NOT test-failure.md' clause from case-003 notes (judges must never penalize extra reference reads). Addresses review threads from coderabbitai and cblecker on #604. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(prow-job-analysis): address review feedback (python floor, metal sno scope, probe error surfacing) - SKILL.md / continue-session.md: correct the Python floor to 3.7+ — prow_job_artifact_search.py uses add_subparsers(required=True), added in 3.7. - metal.md: restrict the sno/single-node trigger to jobs that also carry a metal keyword so cloud SNO jobs (e.g. e2e-aws-ovn-single-node) don't route here; note the SKILL.md routing table is authoritative on conflict. - setup-prow-job-analysis.sh: surface the GCS probe's actual error in the WARN instead of discarding it, to aid debugging in restricted-network environments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(prow-job-analysis eval): soften imperative notes in case-002 and case-004 The `notes` field is context-only per the eval schema, so it should describe the scenario rather than issue MUST/MUST NOT directives to the agent (same fix already applied to case-003 in b21a3f5). Rewrite case-002's "agent MUST recognize … MUST NOT blame" and case-004's "agent MUST route to aggregated.md" into descriptive statements of fact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Chai Bot <ship-help-github@redhat.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Stephen Benjamin <stbenjam+ai@redhat.com>
This enable consumers to install reusable pieces from this repo https://www.anthropic.com/news/claude-code-plugins https://anthropic.mintlify.app/en/docs/claude-code/plugin-marketplaces#how-to-configure-team-marketplaces https://anthropic.mintlify.app/en/docs/claude-code/plugins https://claudecodemarketplace.com/marketplace/anthropic-claude-code