Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ export default defineConfig({
{ text: "Retro", link: "/agents/retro" },
{ text: "Prioritize", link: "/agents/prioritize" },
{ text: "Default vs. Custom", link: "/agents/topics/default-vs-custom" },
{ text: "Escalation Ladder", link: "/agents/topics/escalation-ladder" },
],
},
{
Expand Down
2 changes: 2 additions & 0 deletions docs/agents/topics/default-vs-custom.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ for a stated purpose, that override does not make the agent derived.

## See also

- [Escalation ladder](escalation-ladder.md) — prove-it path before
deriving or replacing a core agent
- [Agents reference](../) — default agent documentation and extension points
- [Configuring agent behavior](../../guides/user/customizing-agents.md) — harness
configuration and layered content resolution
Expand Down
194 changes: 194 additions & 0 deletions docs/agents/topics/escalation-ladder.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Escalation ladder for changing agent behavior

> **Status: Proposed.** Level names and evidence-to-escalate criteria are a
> working proposal pending maintainer agreement on
> [#5372](https://github.com/fullsend-ai/fullsend/issues/5372). Treat this as
> guidance under review, not settled project policy.

Before replacing or deriving from a core agent, prove that lighter options
cannot meet your needs. This page defines the escalation path — start at
Level 1 and move up only when you can show why the current level is
insufficient.

This ladder applies to **core agent roles that fullsend already ships**
(triage, code, review, fix, retro, prioritize). If you are building an agent
for a job fullsend does not cover — a release-notes generator, a
compliance checker, a deployment orchestrator — skip ahead to
[Bring Your Own Agent](../../guides/user/bring-your-own-agent.md). There is
no existing default to exhaust first.

## The four levels

```
Level 1 — Configure Least invasive. No code leaves your repo.
Level 2 — Contribute Fix the gap in the default agent for everyone.
Level 3 — Derive Inherit the default, replace identity components.
Level 4 — Replace Build a parallel agent for the same role.
```
Comment thread
ascerra marked this conversation as resolved.

Each level increases maintenance burden. A derived agent inherits upstream
improvements via `base` but must track breaking changes in the parent
harness. A replacement agent inherits nothing — every upstream improvement
must be replicated independently.

## Level 1: Configure the default agent

Use the extension points the default agent was designed for. These keep you
in [configured-default territory](default-vs-custom.md) and require no
changes to the agent's identity-defining fields. Some rows need a thin
`base`-composed harness (still Level 1); others need no harness at all.

| Extension point | What it does | Guide |
|---|---|---|
| `AGENTS.md` | Project-wide instructions for all agents — code style, test commands, architecture rules, domain context. No harness required. | [Customizing with AGENTS.md](../../guides/user/customizing-with-agents-md.md) |
| Repo skills (`.agents/skills/`) | Domain-specific knowledge for individual agents — linting rules, deployment checklists, label glossaries. No harness required. | [Customizing with Skills](../../guides/user/customizing-with-skills.md) |
| Documented env vars | Per-agent tuning knobs (e.g., `REVIEW_FINDING_SEVERITY_THRESHOLD`). No harness required. | Each agent's [reference page](../) |
| `env:` in harness | Add environment variables without changing the agent's identity. Needs a thin `base` harness. | [Harness field reference](../../guides/user/bring-your-own-agent.md#harness-field-reference) |
| `skills:` in harness | Add skills via `base` composition — concatenated with the base agent's skill list. Needs a thin `base` harness. | [Configuring existing agents](../../guides/user/bring-your-own-agent.md#configuring-existing-agents) |
| `plugins:` in harness | Add language-server plugins. Needs a thin `base` harness. | [Harness field reference](../../guides/user/bring-your-own-agent.md#harness-field-reference) |
| `host_files:` in harness | Inject additional files into the sandbox. Needs a thin `base` harness. | [Harness field reference](../../guides/user/bring-your-own-agent.md#harness-field-reference) |
| Sandbox image layers | Base your image on the default, add tools; point the harness `image:` field at it. Needs a thin `base` harness. | [Extending the sandbox image](../../guides/user/customizing-agents.md#extending-the-sandbox-image) |

**Evidence to escalate:** show that no combination of these extension points
produces the behavior you need. Concrete evidence includes:

- A failing test case or eval run where the agent consistently gets the
wrong answer despite correct `AGENTS.md` instructions
- A skill that cannot influence the agent's behavior because the gap is in
the system prompt, not in domain knowledge
- An env var or config knob that does not exist for the behavior you need to
change

## Level 2: Contribute to the default agent

If Level 1 cannot close the gap, check whether the gap is general — would
other users benefit from the same change? If yes, the fix belongs in the
default agent, not in a private fork.

Contributions include:

- **New extension points** — a new env var, a new skill hook, a new
documented configuration knob
- **Prompt improvements** — clarifications, better instructions, additional
decision criteria in the agent definition
- **Script changes** — pre/post script logic that handles a broader set of
cases
- **Skill improvements** — better procedures, additional steps, broader
coverage in a built-in skill

**How to contribute:** file an issue or open a PR against the default agent.
If the change is in the [fullsend-ai/fullsend](https://github.com/fullsend-ai/fullsend)
repo (harness, scripts, skills, agent definitions), contribute there. If the
agent definition or harness lives in
[fullsend-ai/agents](https://github.com/fullsend-ai/agents), contribute to
that repo instead. First-time contributors must be
[vouched](../../../CONTRIBUTING.md) by a maintainer before opening a PR —
unvouched PRs are auto-closed.

**Evidence to escalate:** show that the improvement you need is specific to
your team's workflow and would not benefit other users. Concrete evidence
includes:

- A proposed upstream change that was reviewed and rejected as too
org-specific
- A behavior change that contradicts the default agent's documented charter
(e.g., you need the review agent to auto-merge, but the default review
agent is explicitly read-only)
- A workflow that requires proprietary integrations (e.g., posting to an
internal Slack channel, querying a private API)

## Level 3: Derive from the default agent

Inherit the default harness via `base` and replace only the components that
must differ. This is a [derived agent](default-vs-custom.md) — it tracks
upstream improvements for everything you did not override, but the
components you replaced are now your responsibility.

Derived agents use `base` composition to inherit the default harness and
override identity-defining fields:

```yaml
base: https://raw.githubusercontent.com/fullsend-ai/fullsend/<sha>/internal/scaffold/fullsend-repo/harness/code.yaml#sha256=abc...

# Override identity-defining components:
agent: agents/my-code-agent.md # Custom system prompt
post_script: scripts/post-my-code.sh # Custom post-processing
slug: my-org-code # Custom identity
```

See [Configuring existing agents](../../guides/user/bring-your-own-agent.md#configuring-existing-agents)
for the full pattern and
[Classification by harness field](default-vs-custom.md#classification-by-harness-field)
for which fields cross the derived threshold.

**What you maintain:**

- Your custom system prompt, scripts, or validation loop
- Compatibility with upstream `base` harness changes (field additions,
schema changes, script interface changes)
- Your own testing and evaluation for the overridden components

**Evidence to escalate:** show that `base` inheritance cannot support your
use case. Concrete evidence includes:

- A fundamental incompatibility with the base harness's script interface
(e.g., the base post-script expects an output format your agent does not
produce, and you cannot adapt either side)
- A need to remove or replace a provider the base harness already declares —
`providers:` entries are concatenated (base + child), not subtractive.
Sandbox policy and image can already be fully overridden at Level 3 via a
scalar field replacement

## Level 4: Replace the role entirely

Build a custom agent from scratch that fills the same role as a default
agent. This is the heaviest option — you inherit nothing from upstream and
must maintain the full harness, agent definition, scripts, skills, and
testing independently.

See [Bring Your Own Agent](../../guides/user/bring-your-own-agent.md) for
the end-to-end guide.

**When this is appropriate:**

- The default agent's architecture is fundamentally incompatible with your
requirements (not just its prompt or scripts)
- You have passed through Levels 1–3 and documented why each is
insufficient
- You have the capacity to maintain a parallel implementation long-term

Register your replacement agent with the same role name in `config.yaml` —
config-registered agents take precedence over built-in agents on name
collision. See
[Registering your agent](../../guides/user/bring-your-own-agent.md#registering-your-agent).

## Prove-it checklist

Before creating a derived or replacement agent for a core role, confirm:

- [ ] **Level 1 exhausted.** You tried `AGENTS.md`, repo skills, env vars,
harness `skills:` / `plugins:` / `host_files:`, and documented config
knobs. None of them address the gap.
- [ ] **Level 2 considered.** The improvement is too org-specific to
contribute upstream, or an upstream contribution was proposed and
rejected.
- [ ] **Level 3 evaluated** (if jumping to Level 4). `base` inheritance
cannot support your use case, and you can articulate why.
- [ ] **Evidence documented.** You have specific examples — failed eval
runs, rejected upstream PRs, architectural constraints — not just a
preference for a different approach.
- [ ] **Maintenance plan.** You have a plan for keeping the derived or
custom agent up to date as the default evolves.

## See also

- [Default, derived, and custom agents](default-vs-custom.md) — how to
classify your agent after making changes
- [Bring Your Own Agent](../../guides/user/bring-your-own-agent.md) —
building and registering custom agents
- [Customizing with AGENTS.md](../../guides/user/customizing-with-agents-md.md)
— Level 1: project-wide instructions
- [Customizing with Skills](../../guides/user/customizing-with-skills.md)
— Level 1: agent-specific domain knowledge
- [Customizing agents](../../guides/user/customizing-agents.md) — harness
configuration and layered content resolution
1 change: 1 addition & 0 deletions docs/guides/user/bring-your-own-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -585,4 +585,5 @@ When configured with `FALLBACK_MINT_URL`, the standalone mint serves custom role
- [Configuring with AGENTS.md](customizing-with-agents-md.md) — repo-level instructions for all agents
- [Configuring agent behavior](customizing-agents.md) — harness configurations and layered content resolution
- [Default, derived, and custom agents](../../agents/topics/default-vs-custom.md) — when configuration crosses into custom agent territory
- [Escalation ladder](../../agents/topics/escalation-ladder.md) — prove-it path before deriving or replacing a core agent
- [Standalone mint](../infrastructure/standalone-mint.md) — custom agent roles and identity
5 changes: 3 additions & 2 deletions docs/guides/user/building-custom-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ This guide walks through creating a custom from-scratch agent on a per-repo
fullsend installation.

Before building from scratch, consider whether extending a default agent would
meet your needs. You can use `base` inheritance to start from a default agent's
harness and override only what differs — see
meet your needs. Start with the
[escalation ladder](../../agents/topics/escalation-ladder.md) to exhaust
lighter options first, and see
[Default, derived, and custom agents](../../agents/topics/default-vs-custom.md)
for the distinction and when each approach makes sense.

Expand Down
18 changes: 18 additions & 0 deletions docs/guides/user/customizing-agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,6 +360,23 @@ Each agent role has its own identity, permissions, and purpose:

## Configuration Examples

### Extending the sandbox image

When `host_files` injection is not enough and you need additional packages or
toolchains in the sandbox, build an image that extends the published base and
point your harness `image:` field at it:

```dockerfile
FROM ghcr.io/fullsend-ai/fullsend-sandbox:latest
RUN apt-get update && apt-get install -y --no-install-recommends rustc \
&& rm -rf /var/lib/apt/lists/*
```

Use `ghcr.io/fullsend-ai/fullsend-code:latest` as the parent instead when you
also need the Go toolchain. Then set `image:` in a thin `base`-composed
harness (see [Configuring existing agents](bring-your-own-agent.md#configuring-existing-agents)).
Pin the parent tag to a digest before CI use.

### Adding Executables

The sandbox already has `/sandbox/workspace/bin` on its `PATH`. To make a
Expand Down Expand Up @@ -546,6 +563,7 @@ disables nothing because no agent has that harness name.

- [Bring Your Own Agent](bring-your-own-agent.md) - Building and registering custom agents from scratch
- [Default, derived, and custom agents](../../agents/topics/default-vs-custom.md) - When does configuration cross into derived or custom agent territory?
- [Escalation ladder](../../agents/topics/escalation-ladder.md) - Prove-it path before deriving or replacing a core agent
- [Getting Started](../getting-started/) - Initial setup
- [Bugfix Workflow](bugfix-workflow.md) - How agents work together
- [Standalone Mint](../infrastructure/standalone-mint.md) - Running your own mint with custom agent roles
Loading