Add eval suite for classify-review-comment skill - #528
Merged
openshift-merge-bot[bot] merged 1 commit intoJun 5, 2026
Merged
Conversation
openshift-ci
Bot
added
the
approved
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (46)
✅ Files skipped from review due to trivial changes (25)
🚧 Files skipped from review as they are similar to previous changes (19)
WalkthroughThis PR adds a case-driven evaluation (14 cases) and an eval harness for the code-review:classify-review-comment skill, plus plugin metadata version bumps (0.0.9 → 0.0.10) in manifest and docs. ChangesReview Comment Classification Evaluation
Sequence DiagramsequenceDiagram
participant Client
participant Runner as claude-code
participant Plugin as code-review
participant Judges
participant Output as classification.json
Client->>Runner: run eval with arguments "{comment}"
Runner->>Plugin: invoke classify-review-comment
Plugin->>Output: produce classification.json
Runner->>Judges: validate structure, labels, correctness, calibration, quality
Judges->>Client: return scores and pass/fail
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (9 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
bryan-cox
force-pushed
the
agent-eval-comment
branch
from
June 4, 2026 22:00
f1c54c3 to
778c800
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/openshift/commands/api-review.md (1)
233-233:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winInvestigate incomplete line at end of file.
Line 233 appears to contain only the line number without any content or change marker. This may indicate a formatting issue or incomplete file.
🔍 Verify file integrity
Check if this is intentional EOF formatting or if content is missing:
#!/bin/bash # Check the actual file ending echo "=== Last 5 lines of the file ===" tail -5 plugins/openshift/commands/api-review.md | cat -A echo -e "\n=== File line count ===" wc -l plugins/openshift/commands/api-review.md echo -e "\n=== Check for trailing newlines ===" od -c plugins/openshift/commands/api-review.md | tail -3🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/openshift/commands/api-review.md` at line 233, The file plugins/openshift/commands/api-review.md ends with an incomplete/trailing line (line 233) that appears empty or malformed; open that file, inspect the final lines (around line 233), remove any stray line-number-only text or stray characters, ensure the file ends with a single newline and no extraneous markers, and commit the cleaned EOF so api-review.md has a proper final newline and no incomplete line entries.
🧹 Nitpick comments (1)
plugins/openshift/commands/api-review.md (1)
10-12: 💤 Low valueConsider adding language specifiers to code blocks.
Markdown linting suggests adding language specifiers to code blocks for better rendering and syntax highlighting. While these are usage examples (not executable code), adding
bashorshellidentifiers would improve documentation quality.📝 Suggested improvements
## Synopsis -``` +```bash /openshift:api-review [pr_url]```diff 1. **Review a PR**: - ``` + ```bash /openshift:api-review https://github.com/openshift/api/pull/2145 ```2. **Review local changes**: - ``` + ```bash /openshift:api-review ```Also applies to: 212-214, 218-220
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/openshift/commands/api-review.md` around lines 10 - 12, Update the fenced code blocks showing the example command "/openshift:api-review [pr_url]" (and the other examples with real/empty URLs) to include a language specifier (e.g., bash or shell) after the opening triple backticks so markdown linters and renderers will apply proper syntax highlighting; locate the blocks containing "/openshift:api-review [pr_url]", "/openshift:api-review https://github.com/openshift/api/pull/2145", and "/openshift:api-review" and change their opening fences from ``` to ```bash.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/code-review/evals/eval-classify-review-comment.yaml`:
- Around line 38-53: The case-mode eval cases are missing the required per-test
metadata; update each annotations.yaml under
plugins/code-review/evals/cases/classify-review-comment/*/annotations.yaml (and
similarly under plugins/ci/evals/cases/payload-analysis/*/annotations.yaml) to
include a top-level metadata: block containing token-usage, judge-size, and tier
(with appropriate values for each test), ensuring the metadata is present on
every test case file so --filter-metadata can find them before defaultTest
merging.
---
Outside diff comments:
In `@plugins/openshift/commands/api-review.md`:
- Line 233: The file plugins/openshift/commands/api-review.md ends with an
incomplete/trailing line (line 233) that appears empty or malformed; open that
file, inspect the final lines (around line 233), remove any stray
line-number-only text or stray characters, ensure the file ends with a single
newline and no extraneous markers, and commit the cleaned EOF so api-review.md
has a proper final newline and no incomplete line entries.
---
Nitpick comments:
In `@plugins/openshift/commands/api-review.md`:
- Around line 10-12: Update the fenced code blocks showing the example command
"/openshift:api-review [pr_url]" (and the other examples with real/empty URLs)
to include a language specifier (e.g., bash or shell) after the opening triple
backticks so markdown linters and renderers will apply proper syntax
highlighting; locate the blocks containing "/openshift:api-review [pr_url]",
"/openshift:api-review https://github.com/openshift/api/pull/2145", and
"/openshift:api-review" and change their opening fences from ``` to ```bash.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 108d342d-b00f-4781-bf61-e6e22850b854
📒 Files selected for processing (30)
plugins/code-review/evals/cases/classify-review-comment/case-001-nitpick-style/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-001-nitpick-style/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-002-question-api-design/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-002-question-api-design/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-003-required-change-nil-panic/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-003-required-change-nil-panic/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-004-required-change-install-failure/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-004-required-change-install-failure/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-005-required-change-test-gap/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-005-required-change-test-gap/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-006-suggestion-ci-rebase/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-006-suggestion-ci-rebase/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-007-required-change-process/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-007-required-change-process/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-008-suggestion-ci-override/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-008-suggestion-ci-override/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-009-unclassified-approval/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-009-unclassified-approval/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-010-unclassified-process-dup/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-010-unclassified-process-dup/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-011-required-change-root-cause/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-011-required-change-root-cause/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-012-suggestion-architecture/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-012-suggestion-architecture/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-013-required-change-security/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-013-required-change-security/input.yamlplugins/code-review/evals/cases/classify-review-comment/case-014-coderabbit-logic-bug/annotations.yamlplugins/code-review/evals/cases/classify-review-comment/case-014-coderabbit-logic-bug/input.yamlplugins/code-review/evals/eval-classify-review-comment.yamlplugins/openshift/commands/api-review.md
bryan-cox
force-pushed
the
agent-eval-comment
branch
from
June 4, 2026 22:10
77961ae to
02a4eea
Compare
stbenjam
reviewed
Jun 4, 2026
| Do not wrap the JSON in markdown code fences in the file. | ||
|
|
||
| models: | ||
| judge: claude-sonnet-4-6 |
Member
There was a problem hiding this comment.
agent-eval-harness have reccomended to stick to Opus for judging
Contributor
Author
There was a problem hiding this comment.
Done. Changed judge model to claude-opus-4-6 per agent-eval-harness recommendation.
AI-assisted response via Claude Code
Adds an agent-eval-harness evaluation config and 14 test cases for the code-review:classify-review-comment skill. Cases cover all severity and topic labels from config.json using real-world PR review comment examples. Judges include 6 deterministic checks (valid JSON, valid labels, confidence range, severity/topic correctness, confidence calibration) and 1 LLM quality judge. Initial run with claude-opus-4-6 achieves 100% pass rate on all deterministic judges and 4.86/5.0 on the LLM judge. Gold reference outputs from the initial run are included as reference-classification.json in each test case directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bryan-cox
force-pushed
the
agent-eval-comment
branch
from
June 4, 2026 23:31
02a4eea to
4527cf4
Compare
1 task
Member
|
/test eval-classify-review-comment |
Contributor
Author
|
/test ? |
Contributor
Author
|
/test eval-classify-review-comment |
Member
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bryan-cox, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
3 tasks
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent-eval-harnessevaluation config (eval-classify-review-comment.yaml) for thecode-review:classify-review-commentskillconfig.json, derived from real-world PR review comment examplesInitial run with
claude-opus-4-6: 100% pass rate on all deterministic judges, 4.86/5.0 mean on the LLM quality judge. $7.34 total cost for 14 cases (~$0.52/case).Test plan
agent-eval-harnessplugin:claude plugin install agent-eval-harness@agent-eval-harness-dev/eval-setupto verify environment/eval-run plugins/code-review/evals/eval-classify-review-comment.yamland confirm all judges passeval/runs/classify-review-comment-eval/<run-id>/report.html🤖 Generated with Claude Code
Summary by CodeRabbit
Tests
Documentation
Chores