diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 0b0f3c581b..573f98c511 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -213,6 +213,7 @@ export default defineConfig({ { text: "Retro", link: "/agents/retro" }, { text: "Prioritize", link: "/agents/prioritize" }, { text: "Default vs. Custom", link: "/agents/topics/default-vs-custom" }, + { text: "Escalation Ladder", link: "/agents/topics/escalation-ladder" }, ], }, { diff --git a/docs/agents/topics/default-vs-custom.md b/docs/agents/topics/default-vs-custom.md index 984b6734da..5bffc8fd32 100644 --- a/docs/agents/topics/default-vs-custom.md +++ b/docs/agents/topics/default-vs-custom.md @@ -87,6 +87,8 @@ for a stated purpose, that override does not make the agent derived. ## See also +- [Escalation ladder](escalation-ladder.md) — prove-it path before + deriving or replacing a core agent - [Agents reference](../) — default agent documentation and extension points - [Configuring agent behavior](../../guides/user/customizing-agents.md) — harness configuration and layered content resolution diff --git a/docs/agents/topics/escalation-ladder.md b/docs/agents/topics/escalation-ladder.md new file mode 100644 index 0000000000..76a8ad0607 --- /dev/null +++ b/docs/agents/topics/escalation-ladder.md @@ -0,0 +1,194 @@ +# Escalation ladder for changing agent behavior + +> **Status: Proposed.** Level names and evidence-to-escalate criteria are a +> working proposal pending maintainer agreement on +> [#5372](https://github.com/fullsend-ai/fullsend/issues/5372). Treat this as +> guidance under review, not settled project policy. + +Before replacing or deriving from a core agent, prove that lighter options +cannot meet your needs. This page defines the escalation path — start at +Level 1 and move up only when you can show why the current level is +insufficient. + +This ladder applies to **core agent roles that fullsend already ships** +(triage, code, review, fix, retro, prioritize). If you are building an agent +for a job fullsend does not cover — a release-notes generator, a +compliance checker, a deployment orchestrator — skip ahead to +[Bring Your Own Agent](../../guides/user/bring-your-own-agent.md). There is +no existing default to exhaust first. + +## The four levels + +``` +Level 1 — Configure Least invasive. No code leaves your repo. +Level 2 — Contribute Fix the gap in the default agent for everyone. +Level 3 — Derive Inherit the default, replace identity components. +Level 4 — Replace Build a parallel agent for the same role. +``` + +Each level increases maintenance burden. A derived agent inherits upstream +improvements via `base` but must track breaking changes in the parent +harness. A replacement agent inherits nothing — every upstream improvement +must be replicated independently. + +## Level 1: Configure the default agent + +Use the extension points the default agent was designed for. These keep you +in [configured-default territory](default-vs-custom.md) and require no +changes to the agent's identity-defining fields. Some rows need a thin +`base`-composed harness (still Level 1); others need no harness at all. + +| Extension point | What it does | Guide | +|---|---|---| +| `AGENTS.md` | Project-wide instructions for all agents — code style, test commands, architecture rules, domain context. No harness required. | [Customizing with AGENTS.md](../../guides/user/customizing-with-agents-md.md) | +| Repo skills (`.agents/skills/`) | Domain-specific knowledge for individual agents — linting rules, deployment checklists, label glossaries. No harness required. | [Customizing with Skills](../../guides/user/customizing-with-skills.md) | +| Documented env vars | Per-agent tuning knobs (e.g., `REVIEW_FINDING_SEVERITY_THRESHOLD`). No harness required. | Each agent's [reference page](../) | +| `env:` in harness | Add environment variables without changing the agent's identity. Needs a thin `base` harness. | [Harness field reference](../../guides/user/bring-your-own-agent.md#harness-field-reference) | +| `skills:` in harness | Add skills via `base` composition — concatenated with the base agent's skill list. Needs a thin `base` harness. | [Configuring existing agents](../../guides/user/bring-your-own-agent.md#configuring-existing-agents) | +| `plugins:` in harness | Add language-server plugins. Needs a thin `base` harness. | [Harness field reference](../../guides/user/bring-your-own-agent.md#harness-field-reference) | +| `host_files:` in harness | Inject additional files into the sandbox. Needs a thin `base` harness. | [Harness field reference](../../guides/user/bring-your-own-agent.md#harness-field-reference) | +| Sandbox image layers | Base your image on the default, add tools; point the harness `image:` field at it. Needs a thin `base` harness. | [Extending the sandbox image](../../guides/user/customizing-agents.md#extending-the-sandbox-image) | + +**Evidence to escalate:** show that no combination of these extension points +produces the behavior you need. Concrete evidence includes: + +- A failing test case or eval run where the agent consistently gets the + wrong answer despite correct `AGENTS.md` instructions +- A skill that cannot influence the agent's behavior because the gap is in + the system prompt, not in domain knowledge +- An env var or config knob that does not exist for the behavior you need to + change + +## Level 2: Contribute to the default agent + +If Level 1 cannot close the gap, check whether the gap is general — would +other users benefit from the same change? If yes, the fix belongs in the +default agent, not in a private fork. + +Contributions include: + +- **New extension points** — a new env var, a new skill hook, a new + documented configuration knob +- **Prompt improvements** — clarifications, better instructions, additional + decision criteria in the agent definition +- **Script changes** — pre/post script logic that handles a broader set of + cases +- **Skill improvements** — better procedures, additional steps, broader + coverage in a built-in skill + +**How to contribute:** file an issue or open a PR against the default agent. +If the change is in the [fullsend-ai/fullsend](https://github.com/fullsend-ai/fullsend) +repo (harness, scripts, skills, agent definitions), contribute there. If the +agent definition or harness lives in +[fullsend-ai/agents](https://github.com/fullsend-ai/agents), contribute to +that repo instead. First-time contributors must be +[vouched](../../../CONTRIBUTING.md) by a maintainer before opening a PR — +unvouched PRs are auto-closed. + +**Evidence to escalate:** show that the improvement you need is specific to +your team's workflow and would not benefit other users. Concrete evidence +includes: + +- A proposed upstream change that was reviewed and rejected as too + org-specific +- A behavior change that contradicts the default agent's documented charter + (e.g., you need the review agent to auto-merge, but the default review + agent is explicitly read-only) +- A workflow that requires proprietary integrations (e.g., posting to an + internal Slack channel, querying a private API) + +## Level 3: Derive from the default agent + +Inherit the default harness via `base` and replace only the components that +must differ. This is a [derived agent](default-vs-custom.md) — it tracks +upstream improvements for everything you did not override, but the +components you replaced are now your responsibility. + +Derived agents use `base` composition to inherit the default harness and +override identity-defining fields: + +```yaml +base: https://raw.githubusercontent.com/fullsend-ai/fullsend//internal/scaffold/fullsend-repo/harness/code.yaml#sha256=abc... + +# Override identity-defining components: +agent: agents/my-code-agent.md # Custom system prompt +post_script: scripts/post-my-code.sh # Custom post-processing +slug: my-org-code # Custom identity +``` + +See [Configuring existing agents](../../guides/user/bring-your-own-agent.md#configuring-existing-agents) +for the full pattern and +[Classification by harness field](default-vs-custom.md#classification-by-harness-field) +for which fields cross the derived threshold. + +**What you maintain:** + +- Your custom system prompt, scripts, or validation loop +- Compatibility with upstream `base` harness changes (field additions, + schema changes, script interface changes) +- Your own testing and evaluation for the overridden components + +**Evidence to escalate:** show that `base` inheritance cannot support your +use case. Concrete evidence includes: + +- A fundamental incompatibility with the base harness's script interface + (e.g., the base post-script expects an output format your agent does not + produce, and you cannot adapt either side) +- A need to remove or replace a provider the base harness already declares — + `providers:` entries are concatenated (base + child), not subtractive. + Sandbox policy and image can already be fully overridden at Level 3 via a + scalar field replacement + +## Level 4: Replace the role entirely + +Build a custom agent from scratch that fills the same role as a default +agent. This is the heaviest option — you inherit nothing from upstream and +must maintain the full harness, agent definition, scripts, skills, and +testing independently. + +See [Bring Your Own Agent](../../guides/user/bring-your-own-agent.md) for +the end-to-end guide. + +**When this is appropriate:** + +- The default agent's architecture is fundamentally incompatible with your + requirements (not just its prompt or scripts) +- You have passed through Levels 1–3 and documented why each is + insufficient +- You have the capacity to maintain a parallel implementation long-term + +Register your replacement agent with the same role name in `config.yaml` — +config-registered agents take precedence over built-in agents on name +collision. See +[Registering your agent](../../guides/user/bring-your-own-agent.md#registering-your-agent). + +## Prove-it checklist + +Before creating a derived or replacement agent for a core role, confirm: + +- [ ] **Level 1 exhausted.** You tried `AGENTS.md`, repo skills, env vars, + harness `skills:` / `plugins:` / `host_files:`, and documented config + knobs. None of them address the gap. +- [ ] **Level 2 considered.** The improvement is too org-specific to + contribute upstream, or an upstream contribution was proposed and + rejected. +- [ ] **Level 3 evaluated** (if jumping to Level 4). `base` inheritance + cannot support your use case, and you can articulate why. +- [ ] **Evidence documented.** You have specific examples — failed eval + runs, rejected upstream PRs, architectural constraints — not just a + preference for a different approach. +- [ ] **Maintenance plan.** You have a plan for keeping the derived or + custom agent up to date as the default evolves. + +## See also + +- [Default, derived, and custom agents](default-vs-custom.md) — how to + classify your agent after making changes +- [Bring Your Own Agent](../../guides/user/bring-your-own-agent.md) — + building and registering custom agents +- [Customizing with AGENTS.md](../../guides/user/customizing-with-agents-md.md) + — Level 1: project-wide instructions +- [Customizing with Skills](../../guides/user/customizing-with-skills.md) + — Level 1: agent-specific domain knowledge +- [Customizing agents](../../guides/user/customizing-agents.md) — harness + configuration and layered content resolution diff --git a/docs/guides/user/bring-your-own-agent.md b/docs/guides/user/bring-your-own-agent.md index 51c7904a5b..eb63210e6b 100644 --- a/docs/guides/user/bring-your-own-agent.md +++ b/docs/guides/user/bring-your-own-agent.md @@ -585,4 +585,5 @@ When configured with `FALLBACK_MINT_URL`, the standalone mint serves custom role - [Configuring with AGENTS.md](customizing-with-agents-md.md) — repo-level instructions for all agents - [Configuring agent behavior](customizing-agents.md) — harness configurations and layered content resolution - [Default, derived, and custom agents](../../agents/topics/default-vs-custom.md) — when configuration crosses into custom agent territory +- [Escalation ladder](../../agents/topics/escalation-ladder.md) — prove-it path before deriving or replacing a core agent - [Standalone mint](../infrastructure/standalone-mint.md) — custom agent roles and identity diff --git a/docs/guides/user/building-custom-agents.md b/docs/guides/user/building-custom-agents.md index c07653ec57..39e5b21289 100644 --- a/docs/guides/user/building-custom-agents.md +++ b/docs/guides/user/building-custom-agents.md @@ -9,8 +9,9 @@ This guide walks through creating a custom from-scratch agent on a per-repo fullsend installation. Before building from scratch, consider whether extending a default agent would -meet your needs. You can use `base` inheritance to start from a default agent's -harness and override only what differs — see +meet your needs. Start with the +[escalation ladder](../../agents/topics/escalation-ladder.md) to exhaust +lighter options first, and see [Default, derived, and custom agents](../../agents/topics/default-vs-custom.md) for the distinction and when each approach makes sense. diff --git a/docs/guides/user/customizing-agents.md b/docs/guides/user/customizing-agents.md index ca3666fc52..5d02c3d3dc 100644 --- a/docs/guides/user/customizing-agents.md +++ b/docs/guides/user/customizing-agents.md @@ -360,6 +360,23 @@ Each agent role has its own identity, permissions, and purpose: ## Configuration Examples +### Extending the sandbox image + +When `host_files` injection is not enough and you need additional packages or +toolchains in the sandbox, build an image that extends the published base and +point your harness `image:` field at it: + +```dockerfile +FROM ghcr.io/fullsend-ai/fullsend-sandbox:latest +RUN apt-get update && apt-get install -y --no-install-recommends rustc \ + && rm -rf /var/lib/apt/lists/* +``` + +Use `ghcr.io/fullsend-ai/fullsend-code:latest` as the parent instead when you +also need the Go toolchain. Then set `image:` in a thin `base`-composed +harness (see [Configuring existing agents](bring-your-own-agent.md#configuring-existing-agents)). +Pin the parent tag to a digest before CI use. + ### Adding Executables The sandbox already has `/sandbox/workspace/bin` on its `PATH`. To make a @@ -546,6 +563,7 @@ disables nothing because no agent has that harness name. - [Bring Your Own Agent](bring-your-own-agent.md) - Building and registering custom agents from scratch - [Default, derived, and custom agents](../../agents/topics/default-vs-custom.md) - When does configuration cross into derived or custom agent territory? +- [Escalation ladder](../../agents/topics/escalation-ladder.md) - Prove-it path before deriving or replacing a core agent - [Getting Started](../getting-started/) - Initial setup - [Bugfix Workflow](bugfix-workflow.md) - How agents work together - [Standalone Mint](../infrastructure/standalone-mint.md) - Running your own mint with custom agent roles