Upgrade skillsaw to v0.10.0 and enable promptfoo linting - #471
Merged
Merged
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
openshift-ci
Bot
added
the
approved
Contributor
|
Important Review skippedAuto reviews are limited based on label configuration. 🚫 Review skipped — only excluded labels are configured. (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughBumps Skillsaw to v0.10.0, adds a new Skillsaw rule ( ChangesPromptfoo budget validation + Skillsaw integration
Tooling pins and invocation
Opt-outs, docs, and minor content fixes
Sequence DiagramsequenceDiagram
autonumber
participant Dev as "Developer"
participant GH as "GitHub Actions"
participant Make as "Local Make"
participant Skillsaw as "Skillsaw Runner"
participant Rule as "promptfoo-budget Rule"
participant Budget as "evals/budget.yaml"
participant Tests as "Promptfoo Tests"
participant Report as "Annotations/Console"
Dev->>GH: Push commit (workflow trigger)
Dev->>Make: Run `make lint` locally
Make->>Skillsaw: Launch container (ghcr.io/stbenjam/skillsaw:0.10.0) with `.skillsaw.yaml`
GH->>Skillsaw: Invoke action (pinned v0.10.0)
Skillsaw->>Rule: Load `.skillsaw/promptfoo_budget_rule.py`
Rule->>Budget: Read policy/budget file
Rule->>Tests: Scan promptfoo tests and read metadata
Rule->>Rule: Validate metadata, thresholds, tiers, aggregate costs
Rule->>Report: Emit errors/warnings per test and per-entity budget
Skillsaw->>GH: Return annotations/report
Skillsaw->>Dev: Local `make lint` prints results
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 9 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (9 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.skillsaw.yaml:
- Line 35: Fix the typo in the YAML boolean for the "enabled" key: replace the
invalid value "falsie" with the valid boolean "false" for the enabled field so
the YAML parses and the rule behaves correctly.
In `@AGENTS.md`:
- Line 37: The markdown currently renders the link as inline code: change the
text "**Register all plugins** in
`[.claude-plugin/marketplace.json](.claude-plugin/marketplace.json)`." so that
the link is not wrapped in backticks (e.g., "**Register all plugins** in
[.claude-plugin/marketplace.json](.claude-plugin/marketplace.json)"), ensuring
the .claude-plugin/marketplace.json path becomes a clickable link; update the
same pattern wherever that inline-code link appears.
- Line 34: The link in the line containing "**Follow existing patterns.** Read
`[plugins/hello-world/commands/echo.md](plugins/hello-world/commands/echo.md)`"
is wrapped in backticks which renders it as code; remove the backticks so the
markdown becomes "**Follow existing patterns.** Read
[plugins/hello-world/commands/echo.md](plugins/hello-world/commands/echo.md)" to
make the link clickable while preserving the surrounding text.
In `@plugins/ci/skills/stage-payload-reverts/SKILL.md`:
- Line 109: The example context string shown as Context (use `--context`): `This
PR is causing blocking job failures ({job names}) in the {stream} {architecture}
payload [{payload_tag}]({release_controller_url}).` should be wrapped in quotes
to prevent shell argument splitting; update the SKILL.md example for the
`--context` flag so the entire multi-word message is quoted (matching the
pattern used in the `payload-experimental-reverts` example), ensuring the quoted
string includes the placeholders `{job names}`, `{stream}`, `{architecture}`,
`{payload_tag}`, and `{release_controller_url}` exactly as shown.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f82dbcdd-d746-4281-b076-ed279a445b16
📒 Files selected for processing (8)
.github/workflows/lint-plugins.yml.skillsaw.yaml.skillsaw/plugindocs_rule.py.skillsaw/promptfoo_budget_rule.pyAGENTS.mdMakefileplugins/ci/skills/stage-payload-reverts/SKILL.mdplugins/sosreport/skills/ovs-db-analysis/SKILL.md
💤 Files with no reviewable changes (1)
- .github/workflows/lint-plugins.yml
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/jira/evals/ready-to-solve.yaml`:
- Line 65: Replace the YAML null-valued metadata key with an explicit empty map:
locate the metadata entry in the ready-to-solve test case (the metadata: line in
plugins/jira/evals/ready-to-solve.yaml) and change it from a null-style key to
an explicit empty map (metadata: {}) so it matches other test cases and avoids
schema/validation issues.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a1257040-5957-4e22-b3a3-b9c8fd69a74f
📒 Files selected for processing (1)
plugins/jira/evals/ready-to-solve.yaml
stbenjam
changed the title
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.skillsaw.yaml:
- Around line 95-97: The Skillsaw config loads the custom rule file
.skillsaw/promptfoo_budget_rule.py under custom-rules but never enables it in
the rules section, so the promptfoo budget checks aren't active; add a rules
entry named promptfoo-budget with enabled: true and severity: error under the
existing rules: block in .skillsaw.yaml to match the custom rule (ensure the key
is exactly promptfoo-budget to match the plugin file's documented name).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3674f1d4-9280-4f5d-b721-f4feb45f6d65
📒 Files selected for processing (1)
.skillsaw.yaml
stbenjam
commented
May 14, 2026
Comment on lines
+30
to
+32
| # skillsaw-disable promptfoo-budget | ||
| # skillsaw-disable promptfoo-assertions | ||
| # skillsaw-disable promptfoo-metadata |
Member
Author
There was a problem hiding this comment.
Disabled since the smoke test doesn't need it
Contributor
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: enxebre, stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This was referenced May 14, 2026
This was referenced May 27, 2026
This was referenced Jun 10, 2026
This was referenced Jun 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This upgrades skillsaw to v0.10.0 and:
There are still many rules disabled, but will fix them in follow-ups to keep this small.
Summary by CodeRabbit
New Features
Chores
Documentation
Tests