Skip to content

feat(code-review): add classify-review-comment skill with config-driven labels - #396

Merged
openshift-merge-bot[bot] merged 8 commits into
openshift-eng:mainfrom
bryan-cox:classify-review-comment-skill
Apr 6, 2026
Merged

openshift-merge-bot[bot] merged 8 commits into
openshift-eng:mainfrom
bryan-cox:classify-review-comment-skill

Conversation

@bryan-cox

@bryan-cox bryan-cox commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a classify-review-comment skill for classifying GitHub PR review comments by severity and topic
  • Supports three input modes: single comment text, GitHub comment URL, or full PR URL with batch classification
  • Structured config.json defines all valid classification values, allowed bots, and noise patterns — making classification deterministic
  • Severity taxonomy: nitpick, suggestion, required_change, question, unclassified
  • Topic taxonomy: style, logic_bug, test_gap, api_design, documentation, ci, approval, process, unclassified
  • Bot filtering driven by allowed_bots list in config.json (default: coderabbitai[bot] only)
  • Noise filtering driven by noise_patterns list in config.json
  • Real-world classification examples from openshift/hypershift PRs

Context

The jira-agent performance dashboard tracks AI-generated PRs and their review feedback. This skill enables classifying PR comments by severity and topic to identify patterns in what reviewers catch most often.

Test plan

  • Classify a single comment: "small nit: rename cnt to count"
  • Classify all comments on a real PR: https://github.com/openshift/hypershift/pull/7620
  • Verify noise filtering excludes CI bot comments and slash commands
  • Verify only bots in allowed_bots (config.json) are classified

🤖 Generated with Claude Code

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 26, 2026
@openshift-ci

openshift-ci Bot commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 26, 2026
@coderabbitai

coderabbitai Bot commented Mar 26, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Added a new classify-review-comment skill to the code-review plugin that categorizes GitHub PR review comments by severity and topic. Updated plugin versions across manifest files to reflect this addition and changes to the plugin structure.

Changes

Cohort / File(s) Summary
Plugin Version Updates
docs/data.json, plugins/code-review/.claude-plugin/plugin.json, .claude-plugin/marketplace.json
Updated code-review plugin version entries across manifest files (0.0.5 → 0.0.6 in docs/data.json and 0.0.5 → 0.0.7 in plugin manifest and marketplace).
New Skill: Classify Review Comment
plugins/code-review/skills/classify-review-comment/SKILL.md, plugins/code-review/skills/classify-review-comment/config.json
Added skill specification defining three control paths for classifying review comments: direct text input, single comment via GitHub URL, and bulk PR comments with pagination and noise filtering. Configuration defines allowed bot authors, noise patterns, severity labels (nitpick, suggestion, required_change, question, unclassified), and review topics (style, logic_bug, test_gap, api_design, documentation, ci, approval, process, unclassified) with examples.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Plugin as classify-review-comment<br/>Plugin
    participant GitHub as GitHub API
    participant Filter as Noise Filter
    participant Classifier as Classifier
    participant Config as Config
    participant Output as Output

    User->>Plugin: Path 1: Comment text
    Plugin->>Classifier: Direct classify
    Classifier->>Config: Load severity/topic labels
    Classifier->>Output: severity, topic, rationale
    
    User->>Plugin: Path 2: Comment URL
    Plugin->>GitHub: Fetch single comment
    GitHub-->>Plugin: Comment text
    Plugin->>Classifier: Classify fetched comment
    Classifier->>Config: Load labels
    Classifier->>Output: severity, topic, rationale

    User->>Plugin: Path 3: Classify PR comments
    Plugin->>GitHub: Fetch all PR comments (paginated)
    GitHub-->>Plugin: Comment list
    Plugin->>Filter: Apply noise filters
    Filter->>Config: Load allowed_bots, noise_patterns
    Filter-->>Plugin: Filtered comments
    Plugin->>Classifier: Classify each comment
    Classifier->>Config: Load severity/topic labels
    Classifier-->>Plugin: Per-comment results
    Plugin->>Output: PR summary with aggregated counts
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 8
✅ Passed checks (8 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
No Real People Names In Style References ✅ Passed Comprehensive search of all JSON and Markdown files found no references to real people's names used as style references in plugin commands, skill documentation, example prompts, or configuration files.
No Assumed Git Remote Names ✅ Passed The new skill files do not use hardcoded git remote names like 'origin' or 'upstream.' Instead, they use explicit GitHub API calls via the gh CLI with parameterized owner/repo values extracted from GitHub URLs.
Git Push Safety Rules ✅ Passed The PR adds a new skill for classifying GitHub PR review comments without introducing any git push commands in the modified or added files.
No Untrusted Mcp Servers ✅ Passed The pull request does not introduce any untrusted MCP server installations. The new classify-review-comment skill uses GitHub CLI commands, not MCP servers.
Ai-Helpers Overlap Detection ✅ Passed The PR introduces a new skill with no redundant or overlapping functionality. The only potentially related item, utils:address-reviews, serves a fundamentally different purpose: it addresses/resolves review feedback, whereas classify-review-comment analyzes patterns. The two use different taxonomies and operate on different goals, making them complementary rather than overlapping.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding a new classify-review-comment skill with configuration-driven labels to the code-review plugin.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@bryan-cox
bryan-cox force-pushed the classify-review-comment-skill branch 2 times, most recently from d6105d9 to fb341e7 Compare March 26, 2026 20:11
@bryan-cox bryan-cox changed the title feat(code-review): expand classify-review-comment skill feat(code-review): add classify-review-comment skill Mar 26, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.claude-plugin/marketplace.json (1)

187-192: ⚠️ Potential issue | 🟡 Minor

Version mismatch: code-review plugin version not updated in marketplace.json.

The code-review plugin version remains 0.0.5 here, but plugins/code-review/.claude-plugin/plugin.json was bumped to 0.0.6. This inconsistency should be fixed.

🔧 Proposed fix
     {
       "name": "code-review",
       "source": "./plugins/code-review",
       "description": "Automated code quality review with language-aware analysis for pre-commit verification",
-      "version": "0.0.5"
+      "version": "0.0.6"
     },
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.claude-plugin/marketplace.json around lines 187 - 192, Update the version
string for the entry with "name": "code-review" in marketplace.json to match the
plugin's actual version (bump "version" from "0.0.5" to "0.0.6") so it stays
consistent with plugins/code-review/.claude-plugin/plugin.json; ensure only the
"version" value for that "code-review" object is changed and no other fields are
modified.
🧹 Nitpick comments (2)
plugins/openshift/commands/api-review.md (2)

209-221: Add language specifiers to example code blocks.

The example code blocks at lines 212-214 and 218-220 are missing language specifiers (flagged by markdownlint MD040).

🔧 Proposed fix
 1. **Review a PR**:
-   ```
+   ```text
    /openshift:api-review https://github.com/openshift/api/pull/2145
    ```
    Checks out the PR, runs lint and convention checks, then switches back to your branch.

 2. **Review local changes**:
-   ```
+   ```text
    /openshift:api-review
    ```
    Reviews local changes against upstream master in the current openshift/api clone.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/openshift/commands/api-review.md` around lines 209 - 221, Update the
two example code fences under the "Review a PR" and "Review local changes"
examples to include a language specifier (use "text") by changing the opening
triple-backtick markers to ```text so markdownlint MD040 is satisfied and the
blocks are explicitly labeled as plain text; locate the opening fences in the
"Review a PR" and "Review local changes" sections and replace them accordingly.

10-12: Add language specifier to fenced code block.

Static analysis flagged this code block as missing a language specifier. Since this is a synopsis showing command usage, consider using text or shell.

🔧 Proposed fix
 ## Synopsis
-```
+```text
 /openshift:api-review [pr_url]
</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @plugins/openshift/commands/api-review.md around lines 10 - 12, The fenced
code block showing the command /openshift:api-review [pr_url] lacks a language
specifier; update the markdown fence to include a language (e.g., use ```text or

the opening triple backticks to include the chosen language token).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugins/openshift/commands/api-review.md`:
- Around line 154-165: Remove the stray "thinking" code block in the
api-review.md doc (the fenced block that begins with ```thinking```), since it's
an AI reasoning placeholder; either delete it entirely or replace it with a
concise user-facing note or an HTML comment describing the validation intent
(e.g., that the doc ensures struct fields have comments, optional fields explain
omission behavior, and validation annotations are documented). Ensure the
replacement is plain prose or an HTML comment so readers aren't exposed to
implementation reasoning.

---

Outside diff comments:
In @.claude-plugin/marketplace.json:
- Around line 187-192: Update the version string for the entry with "name":
"code-review" in marketplace.json to match the plugin's actual version (bump
"version" from "0.0.5" to "0.0.6") so it stays consistent with
plugins/code-review/.claude-plugin/plugin.json; ensure only the "version" value
for that "code-review" object is changed and no other fields are modified.

---

Nitpick comments:
In `@plugins/openshift/commands/api-review.md`:
- Around line 209-221: Update the two example code fences under the "Review a
PR" and "Review local changes" examples to include a language specifier (use
"text") by changing the opening triple-backtick markers to ```text so
markdownlint MD040 is satisfied and the blocks are explicitly labeled as plain
text; locate the opening fences in the "Review a PR" and "Review local changes"
sections and replace them accordingly.
- Around line 10-12: The fenced code block showing the command
`/openshift:api-review [pr_url]` lacks a language specifier; update the markdown
fence to include a language (e.g., use ```text or ```shell) so the block becomes
a properly annotated fenced code block (change the opening triple backticks to
include the chosen language token).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f70952db-d8ea-46b6-b417-aaa4c38e6251

📥 Commits

Reviewing files that changed from the base of the PR and between 82fb683 and 4732811.

📒 Files selected for processing (7)
  • .claude-plugin/marketplace.json
  • PLUGINS.md
  • docs/data.json
  • plugins/code-review/.claude-plugin/plugin.json
  • plugins/code-review/skills/classify-review-comment/SKILL.md
  • plugins/openshift/.claude-plugin/plugin.json
  • plugins/openshift/commands/api-review.md

Comment thread plugins/openshift/commands/api-review.md Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
plugins/code-review/skills/classify-review-comment/SKILL.md (1)

160-160: Use standard phrasing in example text.

Consider replacing “needs fixed” with “needs to be fixed” for clarity in the canonical example set.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/code-review/skills/classify-review-comment/SKILL.md` at line 160,
Update the canonical example text in SKILL.md by replacing the phrase "needs
fixed" with the standard phrasing "needs to be fixed" in the example string
"hypershift-jira-solve-ci - this still needs fixed since the code did not get
pushed" so the example uses clear, grammatically correct wording.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugins/code-review/skills/classify-review-comment/SKILL.md`:
- Around line 16-18: The fenced code block containing "labels.json (in the same
directory as this skill)" is missing a language tag which trips markdown
linters; update the backtick fence so it includes a language identifier (e.g.,
change ``` to ```text or ```json) around the block that contains labels.json to
satisfy linting and clarify the content.
- Around line 67-74: The bot-filter rules are contradictory: the rule "filter
any `*[bot]` except `coderabbitai[bot]`" conflicts with the explicit allowance
of `hypershift-jira-solve-ci[bot]`; update the policy in SKILL.md to use an
explicit allowlist/denylist or ordered precedence so behavior is
deterministic—replace the blanket "any `*[bot]` except `coderabbitai[bot]`" with
a clear allowlist that includes `coderabbitai[bot]` and
`hypershift-jira-solve-ci[bot]` (or document that
`hypershift-jira-solve-ci[bot]` is an exception) and remove the ambiguous
wildcard exception to ensure consistent classification logic referenced in the
CI bot notifications and Do classify sections.

---

Nitpick comments:
In `@plugins/code-review/skills/classify-review-comment/SKILL.md`:
- Line 160: Update the canonical example text in SKILL.md by replacing the
phrase "needs fixed" with the standard phrasing "needs to be fixed" in the
example string "hypershift-jira-solve-ci - this still needs fixed since the code
did not get pushed" so the example uses clear, grammatically correct wording.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7fee93dd-d9d5-41f3-bd3b-51a8ed07eaf4

📥 Commits

Reviewing files that changed from the base of the PR and between fb341e7 and 2259165.

📒 Files selected for processing (4)
  • .claude-plugin/marketplace.json
  • plugins/code-review/.claude-plugin/plugin.json
  • plugins/code-review/skills/classify-review-comment/SKILL.md
  • plugins/code-review/skills/classify-review-comment/labels.json
✅ Files skipped from review due to trivial changes (2)
  • .claude-plugin/marketplace.json
  • plugins/code-review/skills/classify-review-comment/labels.json
🚧 Files skipped from review as they are similar to previous changes (1)
  • plugins/code-review/.claude-plugin/plugin.json

Comment thread plugins/code-review/skills/classify-review-comment/SKILL.md Outdated
Comment thread plugins/code-review/skills/classify-review-comment/SKILL.md Outdated

**Before classifying, filter out noise comments** (these carry no review signal):
- Pure slash commands: body starts with `/` followed by a command word (e.g., `/lgtm`, `/test e2e-aws`, `/approve`, `/retest`, `/cc`)
- CI bot notifications: authors like `openshift-ci-robot`, `openshift-ci[bot]`, `cwbotbot`, or any `*[bot]` author except `coderabbitai[bot]`

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this deterministic and check against a list of approved bots. coderabbit should be allowed as the only default


**Do classify** comments from:
- Human reviewers (all comments, including those directing bots)
- `coderabbitai[bot]` (substantive review comments — code issues, suggestions, questions)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, the bot list should used here

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
plugins/code-review/skills/classify-review-comment/SKILL.md (1)

149-160: ⚠️ Potential issue | 🟠 Major

Examples currently conflict with allowlist-based bot filtering.

At Line 149 and Line 154/159, these examples look like comments authored by hypershift-jira-solve-ci, but your rule at Line 67 filters any bot not in allowed_bots (and current labels.json only allowlists coderabbitai[bot]). That makes these examples contradictory to runtime behavior.

Proposed doc fix
-**Comment:** "hypershift-jira-solve-ci - the unit test job is failing and needs fixed"
+**Comment:** "@hypershift-jira-solve-ci the unit test job is failing and needs to be fixed"
 ```json
 {"severity": "required_change", "topic": "test_gap", "rationale": "Unit tests are failing — the bot made code changes without ensuring tests pass"}

-Comment: "hypershift-jira-solve-ci - rebase the PR to fix the konflux issues"
+Comment: "@hypershift-jira-solve-ci rebase the PR to fix the konflux issues"

{"severity": "suggestion", "topic": "ci", "rationale": "PR needs rebasing to resolve CI pipeline issues — classify by the problem (CI), not the recipient (bot)"}

-Comment: "hypershift-jira-solve-ci - this still needs fixed since the code did not get pushed"
+Comment: "@hypershift-jira-solve-ci this still needs to be fixed since the code did not get pushed"

{"severity": "required_change", "topic": "process", "rationale": "Code changes were not committed/pushed — a process failure, not a code issue"}

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

Verify each finding against the current code and only fix it if needed.

In @plugins/code-review/skills/classify-review-comment/SKILL.md around lines 149

  • 160, The examples at Lines 149 and 154/159 in SKILL.md conflict with the
    allowlist rule (see the bot filter referenced as allowed_bots near the rule at
    Line 67 and labels.json allowlist), so update the example comment strings to use
    an @-mention and clearer wording (e.g., change "hypershift-jira-solve-ci -
    rebase the PR..." to "@hypershift-jira-solve-ci rebase the PR..." and
    "hypershift-jira-solve-ci - this still needs fixed..." to
    "@hypershift-jira-solve-ci this still needs to be fixed..."), and adjust the
    last example's classification JSON to reflect topic "process" and severity
    "required_change" as shown in the proposed doc fix; target the examples block
    around the Comment markers at Lines 149/154/159 and the JSON snippets that
    follow.

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>🤖 Prompt for all review comments with AI agents</summary>

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In @plugins/code-review/skills/classify-review-comment/SKILL.md:

  • Around line 149-160: The examples at Lines 149 and 154/159 in SKILL.md
    conflict with the allowlist rule (see the bot filter referenced as allowed_bots
    near the rule at Line 67 and labels.json allowlist), so update the example
    comment strings to use an @-mention and clearer wording (e.g., change
    "hypershift-jira-solve-ci - rebase the PR..." to "@hypershift-jira-solve-ci
    rebase the PR..." and "hypershift-jira-solve-ci - this still needs fixed..." to
    "@hypershift-jira-solve-ci this still needs to be fixed..."), and adjust the
    last example's classification JSON to reflect topic "process" and severity
    "required_change" as shown in the proposed doc fix; target the examples block
    around the Comment markers at Lines 149/154/159 and the JSON snippets that
    follow.

</details>

---

<details>
<summary>ℹ️ Review info</summary>

<details>
<summary>⚙️ Run configuration</summary>

**Configuration used**: Path: .coderabbit.yaml

**Review profile**: CHILL

**Plan**: Pro

**Run ID**: `9533cc07-5a19-47ea-ae5a-4abea71f5914`

</details>

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 225916515dc28e8da5abb804ef9e88cc6b0c3955 and 672f5efc81259db9b21be9517225bcdd24254c1b.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `plugins/code-review/skills/classify-review-comment/SKILL.md`
* `plugins/code-review/skills/classify-review-comment/labels.json`

</details>

<details>
<summary>✅ Files skipped from review due to trivial changes (1)</summary>

* plugins/code-review/skills/classify-review-comment/labels.json

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->

bryan-cox and others added 5 commits April 1, 2026 13:10
Add a skill for classifying GitHub PR review comments by severity
and topic. Supports three input modes: single comment text, GitHub
comment URL, or full PR URL with batch classification.

Severity categories: nitpick, suggestion, required_change, question
Topic categories: style, logic_bug, test_gap, api_design,
documentation, ci, bot_instruction, approval, process

Includes noise filtering guidance and real-world examples from
openshift/hypershift PRs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move severity and topic label definitions from SKILL.md prose into
a structured labels.json file. The skill now instructs the AI to
read labels.json first and select only from the defined values,
making classification more deterministic and consistent.

Also removes bot_instruction topic, updates noise filters for
CodeRabbit walkthroughs/review-skipped, and bumps version to 0.0.7.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… bot filtering

Address PR review comments: replace hardcoded bot names with
references to the allowed_bots and noise_patterns lists in labels.json.
Add text language tag to fenced code block.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
File now contains allowed_bots, noise_patterns, and classification
labels — config.json better describes its broader scope.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
plugins/code-review/skills/classify-review-comment/SKILL.md (1)

82-82: Tighten wording for clarity.

“subject matter” is wordy here; “subject” reads cleaner without changing meaning.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@plugins/code-review/skills/classify-review-comment/SKILL.md` at line 82,
Update the wording in the list item that begins "4. **Select exactly one topic**
— match the comment's subject matter to the topic descriptions and examples" by
replacing "subject matter" with "subject" so the sentence reads "4. **Select
exactly one topic** — match the comment's subject to the topic descriptions and
examples"; locate the exact line containing that text (the list item starting
with "4. **Select exactly one topic**") and make the single-word replacement to
tighten the phrasing.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugins/code-review/skills/classify-review-comment/config.json`:
- Around line 41-87: The "topic" taxonomy array in config.json is missing the
"bot_instruction" entry referenced by the PR contract; add a new object to the
"topic" array with value "bot_instruction", a short description (e.g.,
"Instructions or commands intended for bots/automation"), and representative
examples (e.g., "run codegen", "apply auto-fix", "trigger bot task") so comments
labeled as bot directives map correctly; update the "topic" list alongside
existing entries (refer to the "topic" array and other "value" objects) to
ensure the declared contract and config stay in sync.

In `@plugins/code-review/skills/classify-review-comment/SKILL.md`:
- Around line 37-40: Documentation omits PR review-body comments (URL pattern
`#pullrequestreview-{id}`) and the corresponding GitHub API endpoints; update
the supported URL formats to include
`https://github.com/{owner}/{repo}/pull/{number}#pullrequestreview-{id}` and add
examples showing the review-body fetch endpoints `GET
/repos/{owner}/{repo}/pulls/{number}/reviews/{id}` (single) and `GET
/repos/{owner}/{repo}/pulls/{number}/reviews` (list) with jq selection for
non-empty `.body`, matching the existing issue and inline comment examples so PR
review summaries (approvals/rejections/comments) are covered.

---

Nitpick comments:
In `@plugins/code-review/skills/classify-review-comment/SKILL.md`:
- Line 82: Update the wording in the list item that begins "4. **Select exactly
one topic** — match the comment's subject matter to the topic descriptions and
examples" by replacing "subject matter" with "subject" so the sentence reads "4.
**Select exactly one topic** — match the comment's subject to the topic
descriptions and examples"; locate the exact line containing that text (the list
item starting with "4. **Select exactly one topic**") and make the single-word
replacement to tighten the phrasing.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 811ba332-d7d4-47d9-aec8-a3c266804dfb

📥 Commits

Reviewing files that changed from the base of the PR and between 672f5ef and edbd932.

📒 Files selected for processing (2)
  • plugins/code-review/skills/classify-review-comment/SKILL.md
  • plugins/code-review/skills/classify-review-comment/config.json

Comment thread plugins/code-review/skills/classify-review-comment/config.json
Comment thread plugins/code-review/skills/classify-review-comment/SKILL.md
@bryan-cox
bryan-cox force-pushed the classify-review-comment-skill branch from edbd932 to 264b8e0 Compare April 1, 2026 17:14
@bryan-cox bryan-cox changed the title feat(code-review): add classify-review-comment skill feat(code-review): add classify-review-comment skill with config-driven labels Apr 1, 2026
Add #pullrequestreview-{id} URL format and /pulls/reviews API
endpoint for fetching review summaries (approvals, rejections,
general comments with state).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bryan-cox
bryan-cox marked this pull request as ready for review April 1, 2026 17:21
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 1, 2026
@wangke19

wangke19 commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Governance Framework Proposal

I wanted to share a governance framework I've developed by working through ideas collaboratively with AI. This is meant as a reference for discussion on how we might approach AI-assisted PR review classification responsibly - not as something prescriptive.


AI PR Review Classification Governance Framework

Version: Final v1.0
Target: openshift-eng/ai-helpers PR review classification workflow
Goal: Human-in-the-loop governance with progressive autonomy, confidence gating, and permanent human override.


1. Executive Summary

This framework defines a production-ready governance model for AI-assisted PR review classification.

The core principle is:

AI assists first, humans govern always.

AI should not replace human engineering judgment at the beginning.
Instead, it should evolve through a structured feedback loop:

Human Decision
    ↓
AI Suggestion
    ↓
Human Correction
    ↓
Experience Capture
    ↓
Rule / Prompt Iteration
    ↓
Confidence Improvement
    ↓
Conditional Delegation

The final objective is graduated autonomy, not blind automation.


2. Governance Principles

2.1 Human-in-the-Loop by Default

All AI-generated PR classifications must initially require human confirmation.

Example:

ai_suggestion:
  label: refactor
  confidence: 0.91
  rationale: "code structure improvement without behavior change"

Reviewer action:

  • Accept
  • Reject
  • Edit
  • Escalate

2.2 Permanent Human Override

Human intervention rights must never be removed.

human_override: true
audit_log: true
rollback_supported: true

This is a non-negotiable engineering safety principle.


2.3 Risk-Based Delegation

Autonomy must be granted by risk level, not average score only.


3. Progressive Autonomy Model


Level 0 — Human Only (Cold Start)

AI provides recommendation only.

AI = advisor
Human = decision maker

Use this phase to collect correction data.


Level 1 — Human Confirm (Recommended Current Phase)

AI proposes labels.

Human must confirm every result.

AI Suggestion -> Human Confirm -> Final Label

This is the best stage for current rollout.


Level 2 — Conditional Autonomy

AI can automatically classify low-risk labels only.

Example:

auto_apply:
  - nit
  - docs
  - style
  - typo
  - naming

manual_required:
  - blocker
  - must-fix
  - security
  - performance
  - breaking-change

This is the recommended delegation model.


Level 3 — AI Primary + Human Override

AI becomes primary classifier.

Human remains escalation and override authority.

AI default execution
Human override always enabled

4. Confidence Gating Model

Confidence score must control workflow routing.

confidence_thresholds:
  auto_apply: 0.95
  human_review: 0.80
  manual_required: 0.00

Routing logic:


Confidence >= 0.95

Auto apply allowed for low-risk labels only.


0.80 <= Confidence < 0.95

Human confirmation required.


Confidence < 0.80

Mandatory manual classification.


5. Human Scoring Framework

Do NOT use a simple 1–10 score only.

Use structured evaluation.

review_score:
  correctness: 5
  severity_assessment: 5
  rationale_quality: 4
  confidence_calibration: 4

Weighted formula:

final_score =
0.4 * correctness +
0.3 * severity +
0.2 * rationale +
0.1 * confidence

Delegation threshold recommendation:

>= 4.5 / 5 for 30 consecutive cases

Only then move to higher autonomy.


6. Experience Learning Framework

Critical requirement:

Store why AI was wrong, not only what was wrong.

Bad example:

ai_label: enhancement
human_label: blocker

Good example:

misclassification_case:
  ai_label: enhancement
  human_label: blocker
  root_cause: "failed to recognize upgrade-path regression risk"
  lesson: "compatibility-related comments should be elevated"

This is the true experience accumulation layer.


7. Feedback Loop Architecture

Human Policy
    ↓
AI Classification
    ↓
Human Correction
    ↓
Error Pattern Mining
    ↓
Rule Update / Prompt Update
    ↓
Model Improvement

This should become a continuous iteration engine.


8. Recommended Production Policy

Recommended immediate rollout:

phase: Level 1

auto_apply:
  - docs
  - nit
  - style

manual_required:
  - security
  - architecture
  - blocker
  - performance

9. Final Recommendation

The correct strategy is:

Human-first → confidence-gated delegation → permanent override

This is the safest and most enterprise-ready AI governance framework for PR review classification.


This framework aligns with the current implementation in this PR, where the skill outputs classifications but doesn't automatically apply them (Level 1). Happy to discuss or revise based on feedback!

…omment

Add a weighted signal rubric (0.00–1.00) for classification confidence,
modeled after the payload analysis scoring approach. Signals include
signal word match, unambiguous category, example pattern match, context
reinforcement, and single viable label. Confidence thresholds gate
workflow routing: >= 0.95 auto-apply, 0.80–0.94 human review, < 0.80
manual classification.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bryan-cox

Copy link
Copy Markdown
Contributor Author

@wangke19 there should be automatic confidence scores applied now.

"examples": ["add a test for empty input", "unit tests failing", "test doesn't validate X"]
},
{
"value": "api_design",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we include architecture design and security?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added architecture_design and security as new topic categories in config.json and SKILL.md with descriptions, signal examples, and real-world classification examples.

…opics

Add two new topic categories per review feedback:
- architecture_design: system architecture, component boundaries, patterns
- security: vulnerabilities, auth issues, input validation, secrets

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@enxebre

enxebre commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Apr 6, 2026
@openshift-ci

openshift-ci Bot commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
openshift-merge-bot Bot merged commit 35cac24 into openshift-eng:main Apr 6, 2026
5 checks passed
@bryan-cox
bryan-cox deleted the classify-review-comment-skill branch April 6, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants