TRT-2612: Make payload analysis RHCOS-version-aware

.claude-plugin/marketplace.json

@@ -49,7 +49,7 @@
       "name": "ci",
       "source": "./plugins/ci",
       "description": "A plugin to work with OpenShift CI and analyze Prow job results",
-      "version": "0.0.42",
+      "version": "0.0.43",
       "category": "ci",
       "keywords": [
         "prow",

docs/index.html

@@ -487,7 +487,7 @@ <h1>ai-helpers</h1>
     {
       "name": "ci",
       "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-      "version": "0.0.42",
+      "version": "0.0.43",
       "has_readme": true,
       "commands": [
         {

plugins/ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ci",
   "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-  "version": "0.0.42",
+  "version": "0.0.43",
   "author": {
     "name": "github.com/openshift-eng"
   }

plugins/ci/commands/analyze-regression.md

@@ -62,10 +62,10 @@ TOKEN=$(oc whoami -t --context="$DPCR_CONTEXT" 2>/dev/null)
 
 This works because `oc` reads from `~/.kube/config` which is bind-mounted from the host. The token stored in the kubeconfig was obtained when the user previously ran `oc login` to the DPCR cluster on the host. If the token is expired, instruct the user to re-authenticate on the host: `oc login https://api.cr.j7t7.p1.openshiftapps.com:6443`.
 
-1. **Load CI Context**: Read all documentation files in `plugins/ci/docs/` for context on tests, jobs, and CI conventions. These contain important notes on specific test frameworks, job ownership, and debugging guidance that should inform the analysis.
+1. **Load CI Context**: Read all documentation files in `plugins/ci/references/` for context on tests, jobs, and CI conventions. These contain important notes on specific test frameworks, job ownership, and debugging guidance that should inform the analysis.
 
    ```bash
-   ls plugins/ci/docs/
+   ls plugins/ci/references/
    ```
 
    Read each file found. Keep this context in mind throughout the analysis — it may affect how you interpret failure patterns, who to recommend contacting, or what the root cause is likely to be.

plugins/ci/references/jobs.md

@@ -0,0 +1,58 @@
+# Job Pattern Reference
+
+When analyzing regressions, use these patterns to identify job types from job names and determine ownership.
+
+## ROSA Classic
+
+- **Match**: job name contains `rosa-sts-ovn`
+- **Example**: `periodic-ci-openshift-release-master-nightly-4.22-e2e-rosa-sts-ovn`
+- **Owner**: HCM OCP Release Enablement
+- **Contact**: `#wg-hcm-ocp-release-enablement` on Slack
+- **Notes**: ROSA (Red Hat OpenShift Service on AWS) classic managed platform jobs.
+
+## Upgrade Jobs
+
+Most blocking jobs install a fresh cluster and run tests. Upgrade jobs are different — they install a cluster at one OCP version and then upgrade it to another. Three upgrade types exist:
+
+- **Micro upgrade**: installs an earlier build of the **same** minor version as the payload, then upgrades to the payload build (e.g., older 5.0 → newer 5.0).
+- **Minor upgrade**: installs the **previous minor** version within the same major, then upgrades (e.g., 4.21 → 4.22).
+- **Major upgrade**: installs the **previous major** OCP version and upgrades to the payload version (e.g., 4.x → 5.0).
+
+Determine the upgrade type from the job name — it will indicate the type (e.g., `major`, `micro`) or the source version being upgraded from. If a job has `upgrade` in its name but no further qualifier, examine the context to determine the type.
+
+The distinction matters because the **install-time OCP version** determines the initial cluster state (RHCOS version, default feature gates, etc.), not the payload/target version. For major upgrades against a 5.x payload, the cluster initially runs OCP 4.x.
+
+## RHCOS Versions
+
+OpenShift clusters run Red Hat Enterprise Linux CoreOS (RHCOS). Two variants exist:
+
+- **RHCOS 9** — based on RHEL 9. The long-standing default for all OCP 4.x releases.
+- **RHCOS 10** — based on RHEL 10. **GA in OCP 5.0.** Has a different kernel, systemd, SELinux policy, and package set than RHCOS 9.
+
+### Detecting RHCOS version from job names
+
+Job names may contain fragments that indicate which RHCOS variant the cluster uses. Check in this order (first match wins):
+
+1. **`rhcos9_10`** — heterogeneous cluster: mixed RHCOS 9 and RHCOS 10 node pools, or a test that upgrades a node pool's RHCOS version during execution.
+   - Example: `periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-rhcos9_10-upgrade`
+2. **`rhcos10`** — RHCOS 10 only.
+   - Example: `periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ipi-ovn-ipv4-rhcos10`
+3. **`rhcos9`** — RHCOS 9 only (explicit).
+   - Example: `periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-rhcos9`
+4. **No fragment** — default by OCP major version **at install time** (not the payload/target version):
+   - OCP 4.x → RHCOS 9
+   - OCP 5.x → RHCOS 9 (current default, will change to RHCOS 10)
+
+For upgrade jobs, use the **install-time** OCP version (see "Upgrade Jobs" above), not the payload/target version. This matters for major upgrades: a major upgrade job in a 5.x payload installs OCP 4.x, so its RHCOS default follows OCP 4.x rules.
+
+### Analysis implications
+
+Variant isolation (a failure appearing only on one RHCOS variant) is diagnostic context that narrows the root cause to OS-specific changes (kernel, systemd, SELinux, package differences between RHEL 9 and RHEL 10).
+
+## Insights Operator
+
+- **Match**: job name contains `insights-operator`
+- **Example**: `periodic-ci-openshift-insights-operator-release-4.22-periodics-e2e-aws-techpreview`
+- **Owner**: Insights Operator team
+- **Contact**: `#forum-observability-intelligence` on Slack (https://redhat.enterprise.slack.com/archives/CLABA9CHY)
+- **Notes**: These jobs sit outside the normal OCP flows. We monitor them for regressions in component readiness, but failures here are best routed to the Insights team rather than treated as core OCP issues.

plugins/ci/skills/analyze-payload/SKILL.md

@@ -110,6 +110,23 @@ Store the PR data keyed by originating payload tag. These PRs are the **candidat
 
 For each failed blocking job in the **target payload**, launch a **parallel subagent** to investigate the failure. Pass the subagent the final Prow URL **and** all previous attempt URLs from Step 2.
 
+#### RHCOS Version Determination
+
+Before launching subagents, determine the RHCOS version for each failed job. Check the job name for these fragments **in order** (first match wins):
+
+1. Job name contains `rhcos9_10` → **heterogeneous** (mixed RHCOS 9 and RHCOS 10 node pools, or RHCOS upgrade during test)
+2. Job name contains `rhcos10` → **RHCOS 10**
+3. Job name contains `rhcos9` → **RHCOS 9** (explicit)
+4. No fragment → default based on the OCP major version **at install time** (not the payload version):
+   - OCP 4.x → **RHCOS 9 (default)**
+   - OCP 5.x → **RHCOS 9 (default)** (current default; will change to RHCOS 10)
+
+   For upgrade jobs, use the **install-time** OCP version (see "Upgrade Jobs" in [jobs.md](../../references/jobs.md)), not the payload/target version. This matters for major upgrades: a major upgrade job in a 5.x payload installs OCP 4.x, so its RHCOS default follows OCP 4.x rules.
+
+Pass the determined RHCOS version to each subagent in the prompt below.
+
+#### Subagent Prompt
+
 Each subagent should determine whether the failure is an install failure or a test failure by checking the JUnit results (e.g., look for `install should succeed*` test failures), then use the appropriate analysis skill. Almost all blocking jobs install a cluster and then run tests, so the job name alone does not tell you the failure type.
 
 You MUST use the following prompt verbatim (substituting the placeholder values) when launching each subagent. Do NOT paraphrase, shorten, or write your own prompt — the specific instructions below are critical for analysis quality:
@@ -120,6 +137,15 @@ You MUST use the following prompt verbatim (substituting the placeholder values)
 >
 > **Non-aggregated jobs**: **Examine the final attempt first**, then compare with previous attempts to determine whether all retries failed the same way. If retries show different failure modes, note this — it distinguishes consistent regressions from intermittent/infrastructure issues. Consistent failures across all attempts strongly indicate a product regression rather than flakiness.
 >
+> **RHCOS version**: This job's cluster runs on **<rhcos_version>**. <rhcos_context>
+
+Where `<rhcos_version>` is the version determined above, and `<rhcos_context>` is one of:
+- For **RHCOS 9** (explicit or default): "RHCOS 9 is based on RHEL 9 — the standard CoreOS variant for this OCP version."
+- For **RHCOS 10**: "RHCOS 10 is based on RHEL 10 with a different kernel, systemd, SELinux policy, and package versions than RHCOS 9. If the failure involves OS-level components (kernel, bootloader, rpm-ostree, MCO, Ignition), consider whether RHEL 10 differences could be the root cause."
+- For **heterogeneous (rhcos9_10)**: "This is a heterogeneous cluster with both RHCOS 9 and RHCOS 10 nodes. Failures may be specific to one node variant — check whether failing nodes are RHCOS 9 or RHCOS 10 when node-level logs are available."
+
+The prompt then continues with:
+
 > First, check the JUnit results or build log to determine whether this is an install failure (look for `install should succeed: overall` or similar install-related test failures) or a test failure (install passed, specific tests failed).
 >
 > Based on the failure type, use the appropriate skill:
@@ -144,6 +170,7 @@ ANALYSIS_RESULT:
 - underlying_job_name: <for aggregated jobs only, extracted from junit artifacts>
 - retries_consistent: yes|no|no_retries|only_final_examined
 - retry_summary: <brief comparison of failure modes across attempts, e.g. "all 3 attempts failed with same KAS crashloop" or "attempt 1 infra timeout, attempts 2-3 test failure", or "no retries" when there was only a single attempt>
+- rhcos_version: rhcos9|rhcos10|rhcos9_10|rhcos9-default|rhcos10-default
 ```
 
 **Note for aggregated jobs**: Since only the final attempt is examined (retries re-run aggregation only), set `retries_consistent: only_final_examined` and `retry_summary: "Aggregated job — only final attempt examined (retries re-run aggregation only)"`.
@@ -158,6 +185,11 @@ After collecting subagent results, look for patterns across multiple jobs:
 
 - **Same failure across a job family** (e.g., all `techpreview` jobs, all `fips` jobs, all `upgrade` jobs): This often indicates a failure specific to that feature set or configuration. Look at what differentiates that job family (feature gates, install-config options, test parameters).
 - **Same failure across multiple platforms**: This often points to a product bug in shared code, though cross-platform infrastructure issues (e.g., CI platform problems) are also possible.
+- **RHCOS variant isolation**: Check whether any failure's root cause or error pattern appears **only** in jobs of one RHCOS variant and **not** in jobs of the other variant. A failure is "variant-isolated" when:
+  - It appears in one or more RHCOS 10 jobs but in zero RHCOS 9 jobs → `failure_scope: "rhcos10-only"`
+  - It appears in one or more RHCOS 9 jobs but in zero RHCOS 10 jobs → `failure_scope: "rhcos9-only"`
+  - Jobs with `rhcos9_10` (heterogeneous) count toward both variants for this check
+  - Variant isolation is strong diagnostic context — it narrows the root cause to OS-specific changes (kernel, systemd, SELinux, package differences between RHEL 9 and RHEL 10).
 
 When patterns emerge, query Sippy for pass rates of related non-blocking jobs to see if the pattern extends beyond blocking jobs.
 
@@ -309,6 +341,7 @@ The report must include the following sections:
 
 A table showing ALL blocking jobs with columns:
 - Job Name
+- RHCOS (the RHCOS version badge for this job: `rhcos9`, `rhcos10`, `rhcos9_10`, or the default version. Use `badge-rhcos9` / `badge-rhcos10` / `badge-rhcos-mixed` CSS classes. When a failure is variant-isolated, add a `variant-isolated` class to highlight the badge)
 - Status (color-coded: green for passed, red for failed)
 - Streak (how many consecutive payloads it has been failing; "N/A" for passed jobs)
 - History (the failure_pattern across the lookback window, e.g., "F F F S F F", showing most recent first; use color-coded markers — red for F, green for S. Each marker should be a link to that job's Prow URL from that payload, when available from the lookback data)
@@ -323,11 +356,19 @@ For each failed job, a collapsible section containing:
   <summary class="failed-job">
     <span class="job-name">{job_name}</span>
     <span class="badge badge-{new|persistent}">{New Failure|Failing for N payloads}</span>
+    <span class="badge badge-{rhcos9|rhcos10|rhcos-mixed}">{RHCOS 9|RHCOS 10|RHCOS 9+10}</span>
   </summary>
   <div class="job-detail">
     <h4>Prow Job</h4>
     <p><a href="{prow_url}">{prow_url}</a></p>
 
+    <!-- Only include when failure is variant-isolated (see Cross-Job Pattern Recognition) -->
+    <div class="variant-callout">
+      This failure is isolated to RHCOS {version} jobs and does not appear in RHCOS {other_version} jobs,
+      indicating an OS-variant-specific root cause (e.g., kernel, systemd, SELinux, or package differences
+      between RHEL 9 and RHEL 10).
+    </div>
+
     <h4>Failure Analysis</h4>
     <div class="analysis">{analysis_from_subagent}</div>
 
@@ -484,6 +525,11 @@ The HTML must be fully self-contained with embedded CSS. Use a GitHub-inspired d
   .badge-infra { background: rgba(210,153,34,0.2); color: var(--orange); border: 1px solid var(--orange); }
   .badge-pass { background: rgba(63,185,80,0.15); color: var(--green); font-size: 0.75rem; padding: 0.1rem 0.5rem; }
   .badge-fail { background: rgba(248,81,73,0.15); color: var(--red); font-size: 0.75rem; padding: 0.1rem 0.5rem; }
+  .badge-rhcos9 { background: rgba(139,148,158,0.15); color: var(--text-muted); font-size: 0.75rem; }
+  .badge-rhcos10 { background: rgba(188,140,255,0.15); color: var(--purple); font-size: 0.75rem; }
+  .badge-rhcos-mixed { background: rgba(210,153,34,0.15); color: var(--orange); font-size: 0.75rem; }
+  .badge.variant-isolated { border: 1px solid currentColor; }
+  .variant-callout { background: rgba(188,140,255,0.1); border-left: 4px solid var(--purple); padding: 0.75rem 1rem; border-radius: 0 0.3rem 0.3rem 0; margin: 0.75rem 0; font-size: 0.9rem; }
   .card { background: var(--surface); border: 1px solid var(--border); border-radius: 0.5rem; padding: 1.25rem; margin: 1rem 0; }
   .summary-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); gap: 1rem; margin: 1rem 0; }
   .stat { background: var(--surface); border: 1px solid var(--border); border-radius: 0.5rem; padding: 1rem; text-align: center; }