Skip to content

chore(fullsend): change models for agents - #89

Merged
JohnStrunk merged 1 commit into
redhat-et:mainfrom
tumido:ci/fullsend-models
Sep 10, 2026
Merged

chore(fullsend): change models for agents#89
JohnStrunk merged 1 commit into
redhat-et:mainfrom
tumido:ci/fullsend-models

Conversation

@tumido

@tumido tumido commented Sep 10, 2026

Copy link
Copy Markdown
Member

Modeled after: fullsend-ai/fullsend#7116

Differs from the one above by selection of Gemini 3.8 instead of Sonnet for the Fix agent. But I can't prove this is better choice for given task.

fullsend agent list --fullsend-dir .fullsend
NAME        SOURCE
code        (built-in)  [runtime=pi model=xai/grok-4.6]
fix         (built-in)  [runtime=pi model=google-vertex/gemini-3.8-flash]
prioritize  (built-in)  [runtime=pi model=google-vertex/gemini-3.8-flash]
review      (built-in)  [runtime=pi model=sonnet subagents: challenger=xai/grok-4.6 correctness=xai/grok-4.6 docs-currency=google-vertex/gemini-3.8-flash security=xai/grok-4.6 style-conventions=google-vertex/gemini-3.8-flash]
retro       (built-in)  [model=sonnet]
triage      harness/triage.yaml  [model=sonnet]
agent UPSTREAM SUGGESTION: runtime UPSTREAM SUGGESTION: model UPSTREAM SUGGESTION: children this PR
code pi xai/grok-4.6 same
fix claude (repo default) sonnet pi + google-vertex/gemini-3.8-flash
prioritize pi google-vertex/gemini-3.8-flash same
review pi sonnet (orchestrator) correctness, security, challengerxai/grok-4.6; docs-currency, style-conventionsgoogle-vertex/gemini-3.8-flash; risk-assessment, intent-coherence, cross-repo-contracts keep their frontmatter sonnet; security-triage keeps haiku same
retro claude sonnet its children name no persona and pass no model, so on Claude they inherit sonnet from the parent same
triage claude sonnet same

By default all agents use effort: high and that stays the same after this PR. Do we want to change it as well?

Summary by CodeRabbit

  • Enhancements
    • Updated AI agent configurations to provide more consistent behavior across coding, fixing, prioritization, review, and retrospective workflows.
    • Improved model routing for specialized tasks, including review workflows with dedicated supporting agents.
    • Preserved triage functionality while aligning it with the updated configuration.

Signed-off-by: Tom Coufal <tcoufal@redhat.com>
@tumido
tumido requested a review from a team September 10, 2026 14:47
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 546cc2cf-4ab8-40a4-921b-fffb2a438262

📥 Commits

Reviewing files that changed from the base of the PR and between a228c1d and 2386031.

📒 Files selected for processing (1)
  • .fullsend/config.yaml

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The Fullsend configuration now defines a sonnet model alias and explicit settings for code, fix, prioritize, review, retro, and triage agents. The earlier standalone triage declaration was removed.

Changes

Agent configuration

Layer / File(s) Summary
Explicit agent settings
.fullsend/config.yaml
The configuration removes the earlier triage declaration and adds model, runtime, subagent, and source settings for the configured agents.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: johnstrunk

Merge Risk: ⚪ Minimal · up to 23860

This change updates Fullsend agent model assignments while preserving the triage configuration, with no actionable merge-blocking risk identified.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: updating model assignments for Fullsend agents.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@fullsend-ai-review fullsend-ai-review Bot added the risk/low PR risk: low label Sep 10, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: low (1/5)

Details

Single configuration file change by a known contributor with a 31-line diff; all Tier 1 signals are minimal and the Tier 2 elevation from 4 distinct authors is offset by zero regressions, zero reverts, and a 2-day-old last commit, yielding a composite of ~1.3 which rounds to low.

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Medium

  • [logic-error] .fullsend/config.yaml:33 — The roles list includes coder (line 14), but the agents block defines an agent named code (line 33). Every other role-to-agent pairing in this file uses an exact name match (fix/fix, prioritize/prioritize, review/review, retro/retro, triage/triage), and the pre-existing base branch also used an exact match (triage role to triage agent). The code/coder mismatch is the sole exception, strongly suggesting it is unintentional. If fullsend resolves agent config by name match, the model (xai/grok-4.6) and runtime (pi) overrides for the code agent will not apply to the coder role.
    Remediation: Rename the agent to name: coder to match the declared role, or change the role from coder to code. Given that every other entry uses exact name matching, aligning the agent name to the role (name: coder) is the safest fix.

Low

  • [missing-authorization] .fullsend/config.yaml — This PR introduces model diversification (xAI/Grok, Google Vertex) and a model alias override without a linked tracking issue.
    Remediation: Consider opening or referencing a tracking issue documenting the decision to diversify model providers.

Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

Comment thread .fullsend/config.yaml
aliases:
sonnet: claude-sonnet-5
agents:
- name: code

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] logic-error

The roles list includes coder (line 14), but the agents block defines an agent named code (line 33). Every other role-to-agent pairing in this file uses an exact name match (fix/fix, prioritize/prioritize, review/review, retro/retro, triage/triage), and the pre-existing base branch also used an exact match (triage role to triage agent). The code/coder mismatch is the sole exception, strongly suggesting it is unintentional. If fullsend resolves agent config by name match, the model (xai/grok-4.6) and runtime (pi) overrides for the code agent will not apply to the coder role.

Suggested fix: Rename the agent to name: coder to match the declared role, or change the role from coder to code. Given that every other entry uses exact name matching, aligning the agent name to the role (name: coder) is the safest fix.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct according to https://fullsend.sh/docs/guides/user/bring-your-own-agent#minimum-viable-agent

Cite:

role is not the agent's name. The agent's name is name: in its .md; role: selects which GitHub App and permissions the mint issues. On the default (hosted) mint, role: must be one of the built-in roles it serves — triage, coder, review, retro, prioritize, fullsend. Pick the one whose permissions fit what your agent does (a code-writing agent uses role: coder). A made-up role like role: my-agent returns 403 from the mint. To use a new role or your own identity, you need your own mint — see [Custom Agent Identity](https://fullsend.sh/docs/guides/user/custom-agent-identity).

@fullsend-ai-review

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:48 PM UTC · Completed 3:02 PM UTC

Commit: 2386031 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.09

@tumido

tumido commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Fullsend is wrong 😄

@tumido tumido added the please-review Ready for maintainers to review this PR label Sep 10, 2026
@tumido

tumido commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

Open question: do we want to stick with effort: high (default) or change that somewhere as well?

@tumido tumido changed the title chore(fullsend): switch to multiple different models chore(fullsend): change models for agents Sep 10, 2026
@JohnStrunk
JohnStrunk added this pull request to the merge queue Sep 10, 2026
Merged via the queue into redhat-et:main with commit 462b4c3 Sep 10, 2026
74 checks passed
@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #89 — chore(fullsend): change models for agents

Workflow: Human-authored config-only PR (31-line diff to .fullsend/config.yaml). Review agent ran on Claude Opus 4.6 at high effort ($3.09, ~15 min). Posted CHANGES_REQUESTED with one medium-severity finding. PR author dismissed the finding with documentation citation. Two humans approved. PR merged ~43 minutes after opening.

Review quality concern — false positive on config.yaml schema: The review agent's sole substantive finding was a [medium] logic-error claiming the agent name code should match the role name coder. The agent reasoned that since 5 of 6 role/agent-name pairs match, the one that doesn't must be a typo. In reality, role and name are intentionally different concepts in the fullsend config schema — roles are permission scopes issued by the mint, while agent names are internal identifiers. The fullsend documentation explicitly states: "role is not the agent's name... a code-writing agent uses role: coder." The PR author correctly dismissed the finding.

Impact: The false positive triggered a CHANGES_REQUESTED status on a correct PR, required the author to look up and cite documentation to dismiss it, and provided no actionable value. This is the same class of issue as fullsend-ai/agents#762 (review agent not understanding fullsend YAML conventions), but #762 specifically covers harness YAML semantics (scaffold path resolution, forge deep-merge, relative paths). This retro identifies a distinct gap: the review agent also lacks understanding of the top-level config.yaml schema, specifically the roles vs agents[].name distinction.

What went well: The fix agent was correctly not dispatched despite the CHANGES_REQUESTED status (no code to fix — the finding was a false positive). The 6 no-op workflow dispatches (from review submission, human comments, label changes, and approvals) are a known upstream concern consolidated in fullsend-ai/fullsend#5817.

Proposals filed

@fullsend-ai-retro

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 3:32 PM UTC · Completed 3:40 PM UTC

Commit: 2386031 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

please-review Ready for maintainers to review this PR risk/low PR risk: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants