Conversation
openshift-ci
Bot
added
the
approved
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughA new Changescreate-prow-agent skill addition
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels
Suggested reviewers
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (9 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/ci/skills/create-prow-agent/SKILL.md (1)
1-560:⚠️ Potential issue | 🟠 MajorResolve HIGH overlap with PR
#532forcreate-prow-agent
- Open PR
#532(“Skill to make it easier to create Prow-based agents”) already modifiesplugins/ci/skills/create-prow-agent/SKILL.md, indicating HIGH overlap and likely duplication/conflict—coordinate/merge with that PR or adjust scope to avoid overlapping functionality.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/skills/create-prow-agent/SKILL.md` around lines 1 - 560, The SKILL.md for the skill named "create-prow-agent" conflicts with an existing PR `#532` that modifies the same skill; resolve by either coordinating/merging with PR `#532` or scoping/renaming this skill to avoid duplication: inspect PR `#532` changes, then either (A) merge your content into the same SKILL.md (preserve frontmatter name: create-prow-agent, reconcile sections and remove duplicated material, add a brief note referencing PR `#532` at the top), or (B) change this skill’s identity (update the frontmatter name and description in SKILL.md to a unique name, e.g., create-prow-agent-v2 or create-prow-agent-guides, and trim overlapping sections), and ensure the user-invocable flag and internal references match the new name so no further conflicts occur.
🧹 Nitpick comments (1)
plugins/ci/skills/create-prow-agent/SKILL.md (1)
152-154: 💤 Low valueAdd language specifiers to fenced code blocks.
Several fenced code blocks lack language specifiers. Adding them improves syntax highlighting and markdown tooling support.
Suggested fixes
Line 152-154:
-``` +```text 1. Gather data → 2. Analyze with Claude → 3. Report resultsLine 157-159: ```diff -``` +```text 1. Query work items → 2. Solve each item → 3. Code review → 4. Fix review findings → 5. Create PR → 6. NotifyLine 162-164: ```diff -``` +```text 1. Poll until data ready → 2. Snapshot data → 3. Analyze → 4. Report → 5. NotifyLine 226-238: ```diff -``` +```text ci-operator/step-registry/{path}/ ├── {agent-name}-workflow.yaml ├── setup/ │ ├── {agent-name}-setup-ref.yaml │ └── {agent-name}-setup-commands.sh ├── process/ │ ├── {agent-name}-process-ref.yaml │ └── {agent-name}-process-commands.sh └── report/ (optional) ├── {agent-name}-report-ref.yaml └── {agent-name}-report-commands.sh</details> Also applies to: 157-159, 162-164, 226-238 <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@plugins/ci/skills/create-prow-agent/SKILL.mdaround lines 152 - 154, In
SKILL.md several fenced code blocks (the sequences containing "1. Gather data →
2. Analyze with Claude → 3. Report results", "1. Query work items → 2. Solve
each item → 3. Code review → 4. Fix review findings → 5. Create PR → 6. Notify",
"1. Poll until data ready → 2. Snapshot data → 3. Analyze → 4. Report → 5.
Notify", and the ci-operator tree block starting with
"ci-operator/step-registry/{path}/") are missing language specifiers; update
each opening triple-backtick to include a language (e.g., ```text) so the blocks
become fenced with a language specifier for proper syntax highlighting.</details> </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.Outside diff comments:
In@plugins/ci/skills/create-prow-agent/SKILL.md:
- Around line 1-560: The SKILL.md for the skill named "create-prow-agent"
conflicts with an existing PR#532that modifies the same skill; resolve by
either coordinating/merging with PR#532or scoping/renaming this skill to avoid
duplication: inspect PR#532changes, then either (A) merge your content into
the same SKILL.md (preserve frontmatter name: create-prow-agent, reconcile
sections and remove duplicated material, add a brief note referencing PR#532at
the top), or (B) change this skill’s identity (update the frontmatter name and
description in SKILL.md to a unique name, e.g., create-prow-agent-v2 or
create-prow-agent-guides, and trim overlapping sections), and ensure the
user-invocable flag and internal references match the new name so no further
conflicts occur.
Nitpick comments:
In@plugins/ci/skills/create-prow-agent/SKILL.md:
- Around line 152-154: In SKILL.md several fenced code blocks (the sequences
containing "1. Gather data → 2. Analyze with Claude → 3. Report results", "1.
Query work items → 2. Solve each item → 3. Code review → 4. Fix review findings
→ 5. Create PR → 6. Notify", "1. Poll until data ready → 2. Snapshot data → 3.
Analyze → 4. Report → 5. Notify", and the ci-operator tree block starting with
"ci-operator/step-registry/{path}/") are missing language specifiers; update
each opening triple-backtick to include a language (e.g., ```text) so the blocks
become fenced with a language specifier for proper syntax highlighting.</details> --- <details> <summary>ℹ️ Review info</summary> <details> <summary>⚙️ Run configuration</summary> **Configuration used**: Path: .coderabbit.yaml **Review profile**: CHILL **Plan**: Enterprise **Run ID**: `a24f1ab2-3fb1-49d3-9914-b92f20f4150d` </details> <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 7ce876b9ac4d2068e3e21f33132745feb2cf80e5 and 6bef82cf2d463a7da3fa7a2ae7302d4db7317ea8. </details> <details> <summary>📒 Files selected for processing (5)</summary> * `.claude-plugin/marketplace.json` * `.gitattributes` * `docs/index.html` * `plugins/ci/.claude-plugin/plugin.json` * `plugins/ci/skills/create-prow-agent/SKILL.md` </details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
openshift-ci
Bot
added
the
needs-rebase
| 1. Poll until data ready → 2. Snapshot data → 3. Analyze → 4. Report → 5. Notify | ||
| ``` | ||
|
|
||
| Write a design summary capturing: |
There was a problem hiding this comment.
We wanna encourage people (especially if they're gonna run these in prow) to have evals thought about at the design stage, and implemented pre merge.
|
|
||
| #### Step 2.8: Write Custom Skills (If Needed) | ||
|
|
||
| If the design identified gaps in existing ai-helpers skills, help the user write custom skills. Each skill needs: |
There was a problem hiding this comment.
Again on evals here, we may wanna think up some base 'this functions as intended' evals for all prow skills that we ship by default / strongly encourage by default.
|
Addressed @theobarberbany's feedback:
|
Interactive guide that walks developers through building a new Prow-based AI agent: brainstorming the design, then generating step registry refs, commands scripts, and ci-operator configs with a prerequisites checklist for credentials to fill in while the openshift/release PR is in review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…attern exploration - Add prerequisites check: Chai Bot MCP, openshift/release clone, gh CLI - Explore existing agent patterns in openshift/release before writing files - Work directly in openshift/release clone instead of temp .work/ directory - Add references section (Autonomous AI Usage doc, HyperShift docs) - Update payload agent description (Chai Bot handoff, multi-agent system) - Fix GitHub App prereqs to mention app configuration and install-id Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Step 1.6 (Design Evals) requiring evals at design time with baseline smoke, golden-path, and guardrail evals for every agent - Add Step 2.9 (Implement Evals) with promptfoo config template and presubmit job for running evals before merge - Add language specifiers to fenced code blocks - Bump skillsaw from v0.11.4 to v0.13.1 in all GitHub Actions workflows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The skillsaw version update belongs in PR openshift-eng#534, not here. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2f068e6 to
c407589
Compare
openshift-ci
Bot
removed
the
needs-rebase
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: stbenjam The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| Evals must be designed alongside the agent, not added after the fact. Ask the user: | ||
|
|
||
| > How will we verify this agent works correctly? Let's define evals now so they ship with the initial PR. |
There was a problem hiding this comment.
content-weak-language): Weak language (vagueness): 'correctly' — Remove 'correctly' — describe what correct behavior looks like
|
|
||
| 1. **Smoke test** — Does the agent start, authenticate, and complete without errors on a trivial input? | ||
| 2. **Golden-path eval** — Given a known input (e.g., a specific Jira issue, a test PR, a sample payload), does the agent produce the expected output? | ||
| 3. **Guardrail eval** — Does the agent correctly refuse or skip inputs outside its scope? |
There was a problem hiding this comment.
content-weak-language): Weak language (vagueness): 'correctly' — Remove 'correctly' — describe what correct behavior looks like
| | Code generation | Generated code compiles, passes lint, tests pass | | ||
| | Triage/analysis | Output matches human-labeled ground truth on N sample inputs | | ||
| | PR creation | PR has correct base branch, labels, description format | | ||
| | Jira updates | Fields set correctly, no duplicate comments | |
There was a problem hiding this comment.
content-weak-language): Weak language (vagueness): 'correctly' — Remove 'correctly' — describe what correct behavior looks like
|
Lol Claude made a mess of this branch lol, will fix on Monday. |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
plugins/ci/skills/create-prow-agent/SKILL.md (3)
612-626:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winRequire explicit user permission before opening a PR.
Line 614 calls
gh pr createwithout asking the user for permission first. Per coding guidelines, commands and skills must never push without explicit user permission — always ask the user before pushing.Revise Step 2.12 to prompt the user for approval before executing
gh pr create. Example: "Ready to open a PR toopenshift/release? Shall I create it?"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/skills/create-prow-agent/SKILL.md` around lines 612 - 626, Step 2.12 currently calls `gh pr create` without explicit user approval; update the SKILL.md step (the "Review and Open PR" section / Step 2.12) to insert an explicit user prompt before executing the `gh pr create` command (e.g., "Ready to open a PR to `openshift/release`? Shall I create it?") and only proceed to run the `gh pr create` invocation after the user responds affirmatively; ensure the prompt stores the user's consent, do not auto-run or auto-merge, and reaffirm the agent will restrict `--allowedTools`, set `--max-turns` and step timeouts before creating the PR.Source: Coding guidelines
226-244:⚠️ Potential issue | 🟠 Major | ⚡ Quick winDiscover git remotes before checking out branches.
Lines 226–244 instruct the user to clone and check out a branch, but do not verify available remotes. Per coding guidelines, code should never assume git remote names ('origin', 'upstream') without first discovering them via
git remote -vorgit branch -vv.Insert a
git remote -vcall after cloning to confirm the remote configuration, and if needed, help the user configure the correct push remote for the feature branch.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/skills/create-prow-agent/SKILL.md` around lines 226 - 244, The instructions assume remote names without verifying them; after cloning the release repo, run git remote -v to discover configured remotes before creating the feature branch add-{agent-name}, and if the desired push remote is missing or different, configure it and set the branch's upstream (so the subsequent git checkout -b add-{agent-name} and push target are correct). Use the discovered remote name when configuring the push upstream for the add-{agent-name} branch to avoid assuming origin/upstream.Source: Coding guidelines
55-55:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUse
git remote -vto discover remotes before constructing fork/push commands.Line 55 uses
gh repo fork openshift/release --clone --remotewithout first discovering available remotes. Per coding guidelines, code must not assume git remote names like 'origin' or 'upstream'; usegit remote -vorgit branch -vvto discover actual remotes before constructing any git commands.Suggested fix: Add a discovery step before forking, or ensure the user is prompted to select the target remote configuration.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/skills/create-prow-agent/SKILL.md` at line 55, The guidance text currently runs the literal command "gh repo fork openshift/release --clone --remote" without discovering existing git remotes; update SKILL.md to add a discovery step that runs "git remote -v" (or "git branch -vv") first, parse or display the results, prompt the user to choose which remote name to use (or document how to map remotes), and then construct the fork/push command using the selected remote name instead of assuming a default; reference the existing command string "gh repo fork openshift/release --clone --remote" when replacing it so the doc shows the new flow (discover remotes → select remote → run gh repo fork with the chosen remote).Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/lint-plugins.yml:
- Around line 20-23: CI workflows use skillsaw 0.13.1 but local config pins
0.11.4; update the local pins to match CI by changing the .skillsaw.yaml version
value to "0.13.1" and updating the Makefile SKILLSAW_IMAGE variable to
ghcr.io/stbenjam/skillsaw:0.13.1 (ensure any other occurrences of '0.11.4' are
replaced), so all uses: stbenjam/skillsaw@... and version: '...' entries are
consistent at 0.13.1 across configs.
---
Outside diff comments:
In `@plugins/ci/skills/create-prow-agent/SKILL.md`:
- Around line 612-626: Step 2.12 currently calls `gh pr create` without explicit
user approval; update the SKILL.md step (the "Review and Open PR" section / Step
2.12) to insert an explicit user prompt before executing the `gh pr create`
command (e.g., "Ready to open a PR to `openshift/release`? Shall I create it?")
and only proceed to run the `gh pr create` invocation after the user responds
affirmatively; ensure the prompt stores the user's consent, do not auto-run or
auto-merge, and reaffirm the agent will restrict `--allowedTools`, set
`--max-turns` and step timeouts before creating the PR.
- Around line 226-244: The instructions assume remote names without verifying
them; after cloning the release repo, run git remote -v to discover configured
remotes before creating the feature branch add-{agent-name}, and if the desired
push remote is missing or different, configure it and set the branch's upstream
(so the subsequent git checkout -b add-{agent-name} and push target are
correct). Use the discovered remote name when configuring the push upstream for
the add-{agent-name} branch to avoid assuming origin/upstream.
- Line 55: The guidance text currently runs the literal command "gh repo fork
openshift/release --clone --remote" without discovering existing git remotes;
update SKILL.md to add a discovery step that runs "git remote -v" (or "git
branch -vv") first, parse or display the results, prompt the user to choose
which remote name to use (or document how to map remotes), and then construct
the fork/push command using the selected remote name instead of assuming a
default; reference the existing command string "gh repo fork openshift/release
--clone --remote" when replacing it so the doc shows the new flow (discover
remotes → select remote → run gh repo fork with the chosen remote).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6bb1b5b5-b2d2-48d6-bdb0-47172506258f
📒 Files selected for processing (4)
.github/workflows/lint-plugins.yml.github/workflows/lint-review.yml.github/workflows/update-docs.ymlplugins/ci/skills/create-prow-agent/SKILL.md
✅ Files skipped from review due to trivial changes (1)
- .github/workflows/lint-review.yml
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/ci/skills/create-prow-agent/SKILL.md`:
- Around line 412-439: The loop reads into variable item but later uses
uninitialized ITEM_ID and STATUS (and writes to
"${SHARED_DIR}/claude-${ITEM_ID}-tokens.json" and appends to "$STATE_FILE"),
which breaks under set -u; fix by parsing item at loop start (e.g., split item
into ITEM_ID and STATUS or assign ITEM_ID=item and set a default STATUS) so the
rest of the body consistently references the initialized variables (ITEM_ID,
STATUS, SHARED_DIR, STATE_FILE) before they are used; ensure any variables
referenced in the grep/jq output path and the echo state line are defined for
every iteration.
- Around line 584-589: Update the checklist to stop instructing users to store a
generated/installation token in the vault and instead only persist the GitHub
App's private-key and installation-id; remove the bullet that says "Store the
app's `private-key` and `installation-id` (one per org) in your vault secret,
along with a generated token for authentication" and replace it with guidance to
persist only `private-key` and `installation-id` and to mint installation tokens
at runtime (use the existing "commands script generates JWT tokens at runtime
using RS256 signing" behavior) so short-lived installation tokens are not
persisted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 7f2e8889-2d27-4010-a64b-a90c9f6bd8e1
📒 Files selected for processing (5)
.claude-plugin/marketplace.json.gitattributesdocs/index.htmlplugins/ci/.claude-plugin/plugin.jsonplugins/ci/skills/create-prow-agent/SKILL.md
✅ Files skipped from review due to trivial changes (4)
- .gitattributes
- plugins/ci/.claude-plugin/plugin.json
- .claude-plugin/marketplace.json
- docs/index.html
openshift-ci
Bot
added
the
needs-rebase
…kill # Conflicts: # .claude-plugin/marketplace.json # docs/index.html # plugins/ci/.claude-plugin/plugin.json
openshift-ci
Bot
removed
the
needs-rebase
- Replace 'correctly' with specific descriptions (lines 179, 185, 194) - Initialize ITEM_ID and STATUS from loop variable before use - Clarify that only private-key and installation-id should be stored in vault, not short-lived generated tokens Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/ci/skills/create-prow-agent/SKILL.md (1)
35-41: 📐 Maintainability & Code Quality | 🔴 CriticalRemove undocumented Chai Bot MCP server reference or document it as officially approved.
The recommendation to enable "Chai Bot MCP server" (lines 35-41) references an external Slack app without documenting it as an officially approved/trusted tool. This violates the requirement that MCP server installations must come from trusted sources with justification and security review. Either remove this recommendation, replace it with a documented and approved alternative, or add explicit approval documentation to the repository establishing Chai Bot as an official OpenShift internal tool.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/ci/skills/create-prow-agent/SKILL.md` around lines 35 - 41, Remove the entire "Chai Bot MCP" section (from the heading through the recommendation block about enabling the Chai Bot MCP server in the Home Page of Slack) unless you have official approval documentation. If Chai Bot is intended to remain as a recommended tool, you must add explicit documentation to the repository establishing it as an officially approved OpenShift internal tool with security review justification, then update the recommendation to reference that approval documentation. The current reference to an external Slack app without documented approval violates MCP server installation requirements.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@plugins/ci/skills/create-prow-agent/SKILL.md`:
- Around line 35-41: Remove the entire "Chai Bot MCP" section (from the heading
through the recommendation block about enabling the Chai Bot MCP server in the
Home Page of Slack) unless you have official approval documentation. If Chai Bot
is intended to remain as a recommended tool, you must add explicit documentation
to the repository establishing it as an officially approved OpenShift internal
tool with security review justification, then update the recommendation to
reference that approval documentation. The current reference to an external
Slack app without documented approval violates MCP server installation
requirements.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: abe89520-3251-43cf-b47c-852b88191e88
📒 Files selected for processing (1)
plugins/ci/skills/create-prow-agent/SKILL.md
|
|
||
| 2. **HyperShift Jira Agent** ([`hypershift/jira-agent/`](https://github.com/openshift/release/tree/main/ci-operator/step-registry/hypershift/jira-agent)) — Runs on a cron schedule. Queries Jira for issues labeled `issue-for-agent`, processes each through a four-phase pipeline (solve → code review → fix → PR creation), tracks state via Jira labels, and uses GitHub App tokens with separate fork/upstream installations. | ||
|
|
||
| 3. **HyperShift Review Agent** ([`hypershift/review-agent/`](https://github.com/openshift/release/tree/main/ci-operator/step-registry/hypershift/review-agent)) — Runs periodically on weekdays. Finds agent-created PRs with unresolved review threads, rebases stale branches, addresses reviewer feedback via Claude, and fixes failing CI checks. Uses a comment analyzer to prevent duplicate bot responses. |
There was a problem hiding this comment.
address review runs only on presubmits atm
hypershift also runs ready-to-solve atm
#618
openshift/release#81553
openshift/release#81820
|
|
||
| #### Chai Bot MCP | ||
|
|
||
| Check if the Chai Bot MCP server is available. If not, strongly recommend it: |
There was a problem hiding this comment.
should this mcp come as dep of this plugin?
| > To enable it, go to the **Home Page** of the Chai Bot app in Slack — it will give you a copy-paste command to configure the MCP server. | ||
|
|
||
| #### openshift/release Clone | ||
|
|
There was a problem hiding this comment.
does it make sense for any of this bash scripting to be injected as dynamic context https://code.claude.com/docs/en/skills#inject-dynamic-context?
|
|
||
| - `/jira:solve` — Analyze a Jira issue and create a PR to solve it | ||
| - `/code-review:pre-commit-review` — Review code changes before committing | ||
| - `/utils:address-reviews` — Address reviewer feedback on a PR |
There was a problem hiding this comment.
this one needs updating, might want to also point to openshift-developer plugin?
|
|
||
| **Multi-phase (code changes):** | ||
| ```text | ||
| 1. Query work items → 2. Solve each item → 3. Code review → 4. Fix review findings → 5. Create PR → 6. Notify |
There was a problem hiding this comment.
fwiw I would want the solve pipeline to explore running in a single claude session that enforce this steps as impl details via workflows, subagents... to alleviate the friction that running multiple sessions introduce for infra and higher level tooling for things like evals, otel
| | PR creation | PR has correct base branch, labels, description format | | ||
| | Jira updates | Fields match expected values, no duplicate comments | | ||
|
|
||
| Use [promptfoo](https://www.promptfoo.dev/) for eval harness. Define eval cases in a `promptfooconfig.yaml` alongside the agent's step registry files. The eval suite should run in a presubmit job against the agent's own PR so regressions are caught before merge. |
There was a problem hiding this comment.
can we please point to eval-harness instead, and all the examples we have in this repo and o/release for running them
| Before writing any files, explore the `openshift/release` clone to understand current patterns for AI-based agents. Read the step registry refs, commands scripts, and workflow YAMLs of existing agents: | ||
|
|
||
| ```bash | ||
| # Find all AI agent step registry entries |
There was a problem hiding this comment.
would we want to have all generic steps in a common step-registry/asdlc
This made some steps in that direction openshift/release#80927
|
|
||
| The step registry path determines the job's identity. Convention: `{team-or-component}/{agent-name}/`. For example: | ||
|
|
||
| - `hypershift/jira-agent/` |
There was a problem hiding this comment.
To remove friction and facilitate experimentation with the solve pipeline and learn what it takes to share an agent across more than one team I would want to have something like openshift/claude/solve instead of hypershift/jira-agent/, and then opt in is just add a label to your jira ticket
Main barrier for that is gh creds I think.
openshift-ci
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
No plans to keep working on this, but it could be a foundation if someone else wants to pick it up. I think Alberto's done a good job about making it easy to onboard a Jira solve agent, which is people's primary use case. |
4 participants
Interactive guide that walks developers through building a new Prow-based
AI agent: brainstorming the design, then generating step registry refs,
commands scripts, and ci-operator configs with a prerequisites checklist
for credentials to fill in while the openshift/release PR is in review.
Summary by CodeRabbit
Release Notes
New Features
Documentation