Skip to content

docs(#5372): add escalation ladder for core agent changes - #5373

Merged
ascerra merged 4 commits into
mainfrom
agent/5372-escalation-ladder
Aug 9, 2026
Merged

docs(#5372): add escalation ladder for core agent changes#5373
ascerra merged 4 commits into
mainfrom
agent/5372-escalation-ladder

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

Add a prove-it escalation ladder that defines a structured path for teams to follow before deriving or replacing a core agent role (triage, code, review, fix, retro, prioritize). The ladder has four levels — configure, contribute, derive, replace — each with concrete evidence requirements for escalation.

Related Issue

Closes #5372

Changes

  • New doc page at docs/agents/topics/escalation-ladder.md with four escalation levels, evidence requirements at each step, and a prove-it checklist
  • Cross-links added from default-vs-custom.md, bring-your-own-agent.md, customizing-agents.md, and building-custom-agents.md
  • Sidebar entry added under the Agents section in VitePress config

Testing

  • Link check passes (lychee --offline: 22 total, 0 errors)
  • Secret scan passes
  • Gitlint passes
  • make lint — pre-commit could not run in sandbox (network error); post-script runs it authoritatively

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • Commits are signed off (DCO) — autonomous agent, exempt
  • I wrote this contribution myself and can explain all changes in it

Closes #5372

Post-script verification

  • Branch is not main/master (agent/5372-escalation-ladder)
  • Secret scan passed (gitleaks — bf051001ad0ddc22d58e95d9d9b46d57e62e5b97..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

Add a prove-it escalation ladder that guides teams through
least-invasive options before deriving or replacing a core agent role.

The ladder defines four levels:
1. Configure — AGENTS.md, skills, env vars, harness extension points
2. Contribute — upstream the improvement to the default agent
3. Derive — inherit via base, replace identity-defining components
4. Replace — build a parallel agent from scratch

Each level documents what evidence is needed to escalate. A prove-it
checklist summarizes the requirements before creating derived or
replacement agents.

Cross-links added from default-vs-custom, bring-your-own-agent,
customizing-agents, and building-custom-agents pages. Sidebar entry
added under the Agents section in the VitePress config.

Note: pre-commit could not run in sandbox (network error fetching
hook repos). The post-script runs pre-commit authoritatively.

Closes #5372
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner July 20, 2026 17:43
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 20, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 20, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:44 PM UTC · Completed 5:55 PM UTC
Commit: 7e0604a · View workflow run →

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown

Site preview

Preview: https://892d39bd-site.fullsend-ai.workers.dev

Commit: 62217e7b94a267210e3517a2fbc0d6b5dab83a2d

@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 20, 2026

Copy link
Copy Markdown

Looks good to me

Previous run

Looks good to me


Labels: PR modifies only documentation files under docs/ and website config

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge component/docs User-facing documentation labels Jul 20, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated 3-agent review-squad pass (claude-coder, claude-researcher, grok). 5 findings survived verification and cross-checking against the repo's own docs (1 HIGH, 4 MEDIUM) — no CRITICAL findings, and link/anchor integrity across the new page is otherwise clean. See inline comments for details.

Comment thread docs/agents/topics/escalation-ladder.md Outdated
Comment thread docs/agents/topics/escalation-ladder.md Outdated
Comment thread docs/agents/topics/escalation-ladder.md
Comment thread docs/agents/topics/escalation-ladder.md Outdated
Comment thread docs/agents/topics/escalation-ladder.md Outdated
Correct Level 3 merge-rule evidence, clarify Level 1 harness vs
identity, mark the page Proposed pending #5372 agreement, note the
vouch gate for Level 2 contributions, and document extending the
sandbox image for the Level 1 link target.

Signed-off-by: Adam Scerra <ascerra@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 4, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 8:10 PM UTC · Ended 8:28 PM UTC
Commit: c9ed4f3 · View workflow run →

Resolve customizing-agents.md conflict: keep main's Configuration
Examples rename and fix-role note, retain Extending the sandbox image
subsection from the review fixes.

Signed-off-by: Adam Scerra <ascerra@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 4, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 8:29 PM UTC · Completed 8:42 PM UTC
Commit: d61ad0f · View workflow run →

@ascerra
ascerra added this pull request to the merge queue Aug 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Aug 6, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 9, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:14 PM UTC · Completed 1:32 PM UTC

Commit: 62217e7 · View workflow run →

@ascerra
ascerra added this pull request to the merge queue Aug 9, 2026
Merged via the queue into main with commit ede081d Aug 9, 2026
14 checks passed
@ascerra
ascerra deleted the agent/5372-escalation-ladder branch August 9, 2026 13:25
@fullsend-ai-retro

fullsend-ai-retro Bot commented Aug 9, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 1:28 PM UTC · Completed 1:46 PM UTC

Commit: 62217e7 · View workflow run →

@fullsend-ai-review

Copy link
Copy Markdown

Review skipped — this PR is already merged.

The /fs-review command only reviews open pull requests.

Posted by fullsend post-review check

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #5373 — Escalation ladder docs

Timeline: Issue #5372 was filed by ascerra on Jul 20. Triage (run 29763872347) and code (run 29764233222) agents processed it within 14 minutes, producing PR #5373 with a new escalation-ladder.md page and cross-links in 4 existing docs. The review agent (run 29764769974) ran 4 sub-agents (correctness/opus, style/sonnet, intent/sonnet, docs-currency/sonnet), all returned 0 findings, and the agent APPROVED with "Looks good to me." Two days later (Jul 22), waynesun09 posted a 3-agent review squad analysis that found 5 real issues (1 HIGH, 4 MEDIUM) — all later confirmed valid when ascerra fixed them manually on Aug 4 (commit c9ed4f3, co-authored with Cursor). The PR merged Aug 9 after a second automated approval and human sign-off from ascerra.

Key finding: review quality gap. The review agent approved the PR while missing all 5 substantive findings the external review squad caught. The correctness sub-agent verified structural properties (anchors exist, identifiers match, specific technical claims match code) but did not verify semantic consistency within the new text. Three distinct failure modes:

  1. Anchor resolution without content verification (findings docs: Add agent-compatible code problem document #1, docs: add agent infrastructure problem document #5): The sub-agent confirmed that customizing-agents.md#customization-examples exists as a heading but never read the content under it to verify it covers sandbox image layering. It read bring-your-own-agent.md but grepped for specific patterns rather than reading the merge-rules table that would have revealed the HIGH-severity claim inaccuracy.
  2. No framing-vs-detail consistency check (finding Add problem areas: Tekton pipeline review, migration path, multi-tenancy #2): The Level 1 intro stated "no harness changes at all" while the table immediately below listed 5 extension points requiring harness YAML. The sub-agent verified individual table entries but never cross-checked them against the framing statement.
  3. Procedural completeness gap (finding Use AI to help formalise intent after rapid local prototyping #4): The "How to contribute" section directed readers to open PRs without mentioning the vouch gate from CONTRIBUTING.md. No sub-agent was scoped to check whether procedural guidance is complete relative to repo processes.

Fix agent was not triggered. waynesun09's review was posted as COMMENTED (not CHANGES_REQUESTED), so the dispatch filter silently dropped it. The human fixed the issues manually 13 days later. This is already well-covered by existing issues: fullsend#5967 (dispatch filter), fullsend#3514 (processing human inline comments), and fullsend#3783 (triggerable from review comments).

Evidence for existing issues:

  • agents#265 (cross-reference verified facts): This PR provides concrete evidence — the correctness sub-agent verified link existence but not semantic accuracy of the claims made about the linked content. Findings docs: Add agent-compatible code problem document #1 and docs: add agent infrastructure problem document #5 are textbook cases of misleading documentation narratives that cross-referencing would have caught.
  • agents#528 (verify script/file references are resolvable): Finding docs: add agent infrastructure problem document #5 shows the gap extends beyond resolution to semantic accuracy — the reference resolved but the target content didn't cover the claimed topic.
  • agents#427 (autonomy evidence for docs-only PRs): This PR provides counter-evidence. The review agent is competent at structural/mechanical docs review but missed all 5 findings on a prescriptive document making behavioral claims about system features. Autonomy for docs-only PRs should be qualified by document type.

No resource waste found. The failed dispatch on Aug 6 (run 31119802507) was a transient GitHub Actions CDN outage (503 Service Unavailable during action download). Post-merge review runs were correctly attributed to other PRs; concurrency groups are working as designed.

Agents repo: fullsend-ai/agents at commit c65eba2a790ebfff571a2a00a06fd1ec3524e934.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/docs User-facing documentation ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: escalation ladder before replacing or deriving a core agent

2 participants