CI Plugin Regression Analysis Improvements - #377
Conversation
openshift-ci
Bot
added
the
approved
WalkthroughThis pull request adds a new "Fetch Job Run Summary" skill to the CI plugin (with docs and CLI), bumps plugin version to 0.0.29, integrates the skill into regression analysis docs, and tweaks test-run fetching URL encoding. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant CI_Plugin as "CI Plugin (skill)"
participant Sippy as "Sippy API"
participant AI as "Analysis/Reporter"
User->>CI_Plugin: request fetch-job-run-summary(prow_job_run_id, format)
CI_Plugin->>Sippy: GET /api/job/run/summary?prow_job_run_id=...
Sippy-->>CI_Plugin: JSON job run data
CI_Plugin->>CI_Plugin: extract_error_pattern(), compute pass rate, dominant errors
CI_Plugin-->>User: formatted text|JSON summary
CI_Plugin->>AI: (optional) provide structured JSON for regression analysis/report
AI-->>User: integrated regression context (reports/triage)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 7 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (7 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
plugins/ci/skills/fetch-job-run-summary/SKILL.md (1)
48-50: Add language identifier to fenced code block.The code block showing the URL pattern should have a language identifier for consistent formatting.
📝 Suggested fix
-``` +```text https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job-name>/<job_run_id></details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@plugins/ci/skills/fetch-job-run-summary/SKILL.mdaround lines 48 - 50,
Update the fenced code block in SKILL.md that shows the Prow URL pattern so it
includes a language identifier; change the opening fence fromtotext so
the block reads as a plain-text snippet (the URL example near the Prow pattern
https://prow.ci.openshift.org/view/gs/test-platform-results/logs//<job_run_id>).</details> </blockquote></details> <details> <summary>plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py (1)</summary><blockquote> `94-99`: **Unused loop variable `test_name`.** The loop variable `test_name` is not used within the loop body. Use `_` to indicate an intentionally unused variable. <details> <summary>🔧 Suggested fix</summary> ```diff - for test_name, error_msg in test_failures.items(): + for _, error_msg in test_failures.items(): pattern = self.extract_error_pattern(error_msg) error_patterns[pattern] += 1 ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary> ``` Verify each finding against the current code and only fix it if needed. In `@plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py` around lines 94 - 99, The loop in which you build error_patterns unnecessarily declares test_name but never uses it; change the iteration over test_failures to use an unused variable name (e.g., replace "for test_name, error_msg in test_failures.items()" with "for _, error_msg in test_failures.items()") so the intent is clear and linters won't flag the unused variable; this affects the block that calls self.extract_error_pattern(error_msg) and updates error_patterns. ``` </details> </blockquote></details> <details> <summary>plugins/ci/commands/analyze-regression.md (2)</summary><blockquote> `692-697`: **Add language identifier to fenced code block.** The code block should have a language identifier for consistent formatting, even for Jira markup. <details> <summary>📝 Suggested fix</summary> ```diff - ``` + ```text {code} [Jira:"Test Framework"] there should not be mass test failures {code} ``` ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@plugins/ci/commands/analyze-regression.mdaround lines 692 - 697, Update the
fenced code block that wraps the Jira {code} example so it includes a language
identifier; locate the triple-backtick block containing the lines "{code}",
"[Jira:"Test Framework"] there should not be mass test failures", and the
closing "{code}" and change the opening fence fromtotext (and keep the
closing fence unchanged) so the snippet is explicitly marked as text for
consistent formatting.</details> --- `939-944`: **Add language identifier to fenced code block.** Same as above — add a language identifier for consistency. <details> <summary>📝 Suggested fix</summary> ```diff - ``` + ```text {code} [Monitor:kubelet-container-restarts][sig-architecture] platform pods in ns/openshift-kube-apiserver should not exit an excessive amount of times {code} ``` ``` </details> <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against the current code and only fix it if needed.
In
@plugins/ci/commands/analyze-regression.mdaround lines 939 - 944, Update the
fenced code block that contains the Jira {code} wrapper and the test name line
"[Monitor:kubelet-container-restarts][sig-architecture] platform pods in
ns/openshift-kube-apiserver should not exit an excessive amount of times" to
include a language identifier by changing the opening triple-backticks to
"text" so the block becomestext, then the {code} block and test name, and
close with ```; ensure the single-line test name remains unchanged inside the
{code} wrapper.</details> </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against the current code and only fix it if needed.
Inline comments:
In@plugins/ci/skills/fetch-job-run-summary/SKILL.md:
- Around line 99-118: The documentation JSON example is inconsistent with the
implementation in fetch_job_run_summary.py: the example shows failures_by_sig
but the code emits a flat failed_tests list and dominant_error_patterns; update
the SKILL.md example to replace the failures_by_sig object with the actual
"failed_tests": [ { "test_name": "...", "error": "..." } ] field (and remove the
failures_by_sig section) so the example matches the output produced by
fetch_job_run_summary.py and references the same keys used by that script.
Nitpick comments:
In@plugins/ci/commands/analyze-regression.md:
- Around line 692-697: Update the fenced code block that wraps the Jira {code}
example so it includes a language identifier; locate the triple-backtick block
containing the lines "{code}", "[Jira:"Test Framework"] there should not be
mass test failures", and the closing "{code}" and change the opening fence from
totext (and keep the closing fence unchanged) so the snippet is
explicitly marked as text for consistent formatting.- Around line 939-944: Update the fenced code block that contains the Jira
{code} wrapper and the test name line
"[Monitor:kubelet-container-restarts][sig-architecture] platform pods in
ns/openshift-kube-apiserver should not exit an excessive amount of times" to
include a language identifier by changing the opening triple-backticks to
"text" so the block becomestext, then the {code} block and test name, and
close with ```; ensure the single-line test name remains unchanged inside the
{code} wrapper.In
@plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py:
- Around line 94-99: The loop in which you build error_patterns unnecessarily
declares test_name but never uses it; change the iteration over test_failures to
use an unused variable name (e.g., replace "for test_name, error_msg in
test_failures.items()" with "for _, error_msg in test_failures.items()") so the
intent is clear and linters won't flag the unused variable; this affects the
block that calls self.extract_error_pattern(error_msg) and updates
error_patterns.In
@plugins/ci/skills/fetch-job-run-summary/SKILL.md:
- Around line 48-50: Update the fenced code block in SKILL.md that shows the
Prow URL pattern so it includes a language identifier; change the opening fence
fromtotext so the block reads as a plain-text snippet (the URL example
near the Prow pattern
https://prow.ci.openshift.org/view/gs/test-platform-results/logs//<job_run_id>).</details> --- <details> <summary>ℹ️ Review info</summary> <details> <summary>⚙️ Run configuration</summary> **Configuration used**: Path: .coderabbit.yaml **Review profile**: CHILL **Plan**: Pro **Run ID**: `f809b17d-9e3d-4dd6-9b7a-985e84acd79c` </details> <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 7b7e2e0f4714f769a92317d6d29681aa7947fb9b and 1afd0863354b90405cbbbce20555bcc7df0ebc3a. </details> <details> <summary>📒 Files selected for processing (8)</summary> * `.claude-plugin/marketplace.json` * `docs/data.json` * `plugins/ci/.claude-plugin/plugin.json` * `plugins/ci/commands/analyze-regression.md` * `plugins/ci/docs/tests.md` * `plugins/ci/skills/fetch-job-run-summary/SKILL.md` * `plugins/ci/skills/fetch-job-run-summary/fetch_job_run_summary.py` * `plugins/ci/skills/fetch-test-runs/fetch_test_runs.py` </details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@plugins/ci/skills/fetch-job-run-summary/SKILL.md`:
- Around line 48-50: The fenced code block in SKILL.md containing the Prow URL
lacks a language identifier; update that block to include a language tag (e.g.,
use ```text) so the snippet is lint-compliant (markdownlint MD040) and remains
displayed as plain text. Locate the fenced block with the URL
"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job-name>/<job_run_id>"
and change the opening fence from ``` to ```text (or another appropriate
language identifier).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1b9311e3-4b01-41f7-8abf-7c6f414627e4
📒 Files selected for processing (1)
plugins/ci/skills/fetch-job-run-summary/SKILL.md
| ``` | ||
| https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job-name>/<job_run_id> | ||
| ``` |
There was a problem hiding this comment.
Add a language identifier to the fenced code block.
The fenced block at Line 48 is missing a language tag (markdownlint MD040).
📝 Suggested fix
-```
+```text
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job-name>/<job_run_id></details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.21.0)
[warning] 48-48: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@plugins/ci/skills/fetch-job-run-summary/SKILL.md` around lines 48 - 50, The
fenced code block in SKILL.md containing the Prow URL lacks a language
identifier; update that block to include a language tag (e.g., use ```text) so
the snippet is lint-compliant (markdownlint MD040) and remains displayed as
plain text. Locate the fenced block with the URL
"https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job-name>/<job_run_id>"
and change the opening fence from ``` to ```text (or another appropriate
language identifier).
|
/test images |
|
/lgtm I think it's OK to merge while the image problems get resolved, images is only used for the agentic Prow jobs |
|
@stbenjam: Overrode contexts on behalf of stbenjam: ci/prow/images DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgoodwin, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
2 participants
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Chores