-
Notifications
You must be signed in to change notification settings - Fork 94
feat(#6581): per-agent runtime, model and effort on agents: entries #6583
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,190 @@ | ||
| --- | ||
| title: "91. Per-agent runtime, model and effort on agents: entries" | ||
| status: Accepted | ||
| relates_to: | ||
| - agent-architecture | ||
| topics: | ||
| - config | ||
| - harness | ||
| - runtime | ||
| --- | ||
|
|
||
| # 91. Per-agent runtime, model and effort on agents: entries | ||
|
|
||
| Date: 2026-08-25 | ||
|
|
||
| ## Status | ||
|
|
||
| Accepted | ||
|
|
||
| ## Context | ||
|
|
||
| `.fullsend/config.yaml` could not express per-agent `model`, `runtime`, or | ||
| `effort`. A repo that needed different agents on different models or | ||
| runtimes had to set GitHub repository variables (`TRIAGE_FULLSEND_MODEL`, | ||
| `CODE_FULLSEND_RUNTIME`, ...), which live outside the repository: no review, | ||
| no git history, no diff, and nothing in the repo records the intent. | ||
|
|
||
| The per-role `model:` field in the harness is per-agent by construction, | ||
| but repos consuming remote harnesses cannot reach it. The `validModelName` | ||
| regex (`^[a-zA-Z0-9_.@-]+$`) also prevented provider-qualified model | ||
| identifiers (e.g. `xai-vertex/xai/grok-4.6`) in harness files, forcing | ||
| repos on pi to use repository variables as the only path. | ||
|
|
||
| `runtime:` was a single repo-wide key (ADR 0033), yet the case that | ||
| motivates this record is precisely a repo whose agents run on different | ||
| runtimes: fullsend-ai/pi-xai-vertex runs triage and review on pi (Grok) | ||
| and code, fix and retro on Claude Code. | ||
|
|
||
| Related issues: #6529 (per-role model + effort, scoped out runtime), | ||
| #6570 (validModelName `/` restriction), #6577 (alias resolution bug, | ||
| independent). #6529 and #6570 are folded into this record deliberately: | ||
| a per-agent model field that cannot name `provider/id` would not serve | ||
| the pi case that motivates it, so the schema and the validation rule are | ||
| one decision, not two. | ||
|
|
||
| ## Options | ||
|
|
||
| **A separate `role_overrides:` map** (the first cut of the implementing | ||
| PR, #6583) — keyed by agent name, with `runtime`/`model`/`effort` per key. | ||
| Rejected before merge: `config.yaml` already has a per-agent place — the | ||
| `agents:` list (ADR 0058) — keyed by the same names with the same layered | ||
| merge, so a second per-agent section would have split "what runs" from | ||
| "how it runs" and carried a misleading name (it was keyed by agent name, | ||
| not harness role, because `code` and `fix` share `role: coder`). | ||
|
|
||
| **Fields on the `agents:` entry** — chosen; see Decision. | ||
|
|
||
| ## Decision | ||
|
|
||
| ### 1. `runtime`, `model`, `effort` on `agents:` entries | ||
|
|
||
| An `agents:` entry may set the three fields. A built-in agent is tuned | ||
| with a **name-only entry** (no `source:`); a custom agent carries them on | ||
| its `source:` entry: | ||
|
|
||
| ```yaml | ||
| runtime: pi # repo default for agents that set none | ||
| agents: | ||
| - name: triage | ||
| model: xai-vertex/xai/grok-4.6 | ||
| - name: code | ||
| runtime: claude | ||
| model: sonnet | ||
| effort: high | ||
| - source: https://raw.githubusercontent.com/acme/agents/<sha>/harness/lint.yaml#sha256=<hash> | ||
| model: haiku # custom agent "lint" (name derived from the file, ADR 0058) | ||
| ``` | ||
|
|
||
| Names are **agent names** as passed to `fullsend run <agent>` (triage, | ||
| code, review, fix, retro, prioritize) or a custom entry's name, matched | ||
| case-insensitively. They are NOT harness `role:` values — `code` and | ||
| `fix` both carry `role: coder`, while the existing role-prefixed | ||
| repository variables (`CODE_FULLSEND_MODEL`, `FIX_FULLSEND_MODEL`) already | ||
| key on agent name. | ||
|
|
||
| An enabled entry without `source:` must name a built-in agent and set at | ||
| least one field; anything else is rejected (`coder` gets a "did you mean | ||
| `code`?" hint). Such an *override-only* entry registers no harness: the | ||
| built-in keeps resolving through the agents-repo fallback, and it is not | ||
| enumerated as a custom harness, locked, or listed as one. | ||
|
|
||
| This narrows ADR 0045's "config.yaml does not gain deep-merge | ||
| capabilities or per-agent override entries": that decision kept an | ||
| agent's *definition* out of `config.yaml`, and it still holds — these | ||
| fields tune three operational knobs of an already resolved harness; they | ||
| do not define or compose one. Anything beyond runtime/model/effort still | ||
| belongs in a harness (`base` composition). | ||
|
|
||
| `fullsend agent set <name> [--runtime] [--model] [--effort]` writes the | ||
| entry so the file need not be edited by hand. | ||
|
|
||
| ### 2. Precedence | ||
|
|
||
| Config-layer addition, slotting in below per-run overrides: | ||
|
|
||
| ``` | ||
| --runtime/--model/--effort flag | ||
| > FULLSEND_* env (including role-prefixed repository variables) | ||
| > the agent's agents: entry | ||
| > repo-wide runtime: / harness model: effort: | ||
| > default | ||
| ``` | ||
|
|
||
| The repo-wide `runtime:` key is kept as the default for agents that set | ||
| none — deliberately not removed: existing configs, `fullsend github setup | ||
| --runtime` and the behaviour-test installs depend on it, and a default is | ||
| still the right shape for a repo where every agent runs the same way. | ||
| Per-agent is now the primary place to select a runtime; the docs steer | ||
| there. | ||
|
|
||
| Entries merge per field across the layered config (ADR 0069): the | ||
| overlay's non-empty value wins, an empty value inherits the base's; there | ||
| is no tombstone to unset a base value short of restating the entry. | ||
|
|
||
| Plan output, stderr `runtime: selected ...` lines, and `metrics.json` | ||
| (`runtime_source`, `override_source`) name the source as | ||
| `<config path> agents.<name>` (the effective config file). | ||
|
|
||
| `fullsend run` does not call `Validate()` on the config it loads, so it | ||
| validates the effective `agents:` list itself on every run — names, | ||
| runtime against `ValidRuntimes()` (a stub runtime cannot be activated | ||
| through an entry any more than through `runtime:`), effort against the | ||
| shared levels, model against the shared model-reference syntax — in the | ||
| overlay and in `config.base.yaml` alike, and fails the run with an error | ||
| naming the file and entry rather than silently running without the | ||
| settings. Runtime selection and the model/effort application read one | ||
| loaded config, so an entry's three fields always agree on their source. | ||
|
|
||
| The effective entries, with these fields, are exposed to overlay CEL | ||
| expressions as `config.agents` (ADR 0088). | ||
|
|
||
| ### 3. Segment-based model validation | ||
|
|
||
| A shared `ValidModelRef` regex (`^[a-zA-Z0-9_.@-]+(/[a-zA-Z0-9_.@-]+)*$`) | ||
| replaces the harness-local `validModelName`. This is a superset of the | ||
| previous rule: existing single-segment model names continue to validate, | ||
| and `provider/id` forms are now accepted in both harness `model:` fields | ||
| and `agents:` entry `model:` values. Malformed forms (`/leading`, | ||
| `trailing/`, `a//b`) are rejected. | ||
|
|
||
| The effort level list moves to `config.ValidEffortLevels()` for the same | ||
| reason: `harness` imports `config`, so the shared lists live in `config` | ||
| and both validators read one source of truth. | ||
|
|
||
| ### 4. Deferred: `models.aliases` | ||
|
|
||
| Provider-qualified per-agent models remove most of the need for the | ||
| `models.aliases` map proposed in #6529. Alias resolution is tracked | ||
| separately in #6577. This ADR does not introduce an alias system. | ||
|
|
||
| ### 5. Scope: per-repo first | ||
|
|
||
| The fields are read wherever `agents:` entries are (org-mode configs | ||
| included, since the list is shared), but the CLI (`agent set`), the setup | ||
| PR text and the docs target per-repo installs; per-org installation mode | ||
| is deprecated (ADR 0044). | ||
|
|
||
| ## Consequences | ||
|
|
||
| - Repos express per-agent runtime, model and effort in one reviewable, | ||
| version-controlled list that also says which harness runs — one place | ||
| per agent. | ||
| - Repository variables remain valid for one-off experiments and for | ||
| repos that prefer out-of-band configuration. | ||
| - The harness `model:` field now accepts `/` in model identifiers, | ||
| unblocking provider-qualified models in harness files. | ||
| - Because `config.yaml` now carries durable per-agent intent, a | ||
| `fullsend github setup` re-run keeps an existing per-repo `config.yaml` | ||
| untouched unless a flag targets a config key, and then changes only | ||
| that key; managed workflow files still refresh. The converge | ||
| full-rescaffold repair (missing workflow) and the GitLab setup path keep | ||
| regenerating the file as before. | ||
| - The reusable workflow's `config.yaml` guard allows an enabled entry | ||
| without `source:` when it sets a field. A repo must bump its pin to a | ||
| version that carries this change before adding such an entry: an older | ||
| pinned workflow rejects it for every agent, whereas a separate unknown | ||
| key would have been ignored. The docs say so. | ||
| - `review` and `retro` can be put on another runtime through their | ||
| entries but are documented to stay on Claude Code today; this is not | ||
| enforced in validation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.