feat: implement triage agent (Story 3, #126) by ralphbean · Pull Request #279 · fullsend-ai/fullsend

ralphbean · 2026-04-20T19:34:49Z

Summary

Implements the triage agent end-to-end: agent prompt, JSON schema validation, pre/post scripts, shim workflow re-entry
Adds ADR 0020 for harness-level output schema enforcement
Three-layer pipeline: agent → schema validation (retry on fail) → post-script (GitHub mutations)
Hybrid questioning strategy with clarity scoring and anti-premature-resolution
Verified working live on appdumpster/test-repo#4 — agent triaged, commented via bot, post-script ran

Key changes

agents/triage.md — Production triage agent prompt with clarity scoring (4 dimensions), anti-premature-resolution rule, structured JSON output
schemas/triage-result.schema.json — JSON Schema with conditional requirements (if/then) for action-specific fields
scripts/pre-triage.sh — Strips triage labels before agent runs (label state machine reset)
scripts/post-triage.sh — Deterministic GitHub mutations (comment + label + close) based on agent JSON output
scripts/validate-output-schema.sh — Generic schema validator for any agent (ADR 0020)
harness/triage.yaml — Updated with pre_script, validation_loop (max 2 iterations), post_script, runner_env with FULLSEND_DIR
templates/shim-workflow.yaml — Triage re-entry on human reply to needs-info issues (bot loop prevention)
internal/cli/run.go — FULLSEND_DIR env injection for ValidateRunnerEnv, script chmod for Contents API compatibility
internal/harness/harness.go — Scripts() method for executable permission management
internal/forge/ — ListIssueComments for triage context gathering

Also includes (cherry-picked from #277)

--vendor-fullsend-binary flag for dev iteration without cutting releases

Test plan

make go-test — all unit tests pass (scaffold, harness, layers, forge, cli)
make go-vet — clean
make lint — clean
Bash test suites: 10/10 schema validation, 6/6 post-triage
Live verification: triage workflow succeeded on appdumpster/test-repo#4 (run)
E2E tests (depend on feat: add --vendor-fullsend-binary for dev iteration #277 merge for vendored binary support, or a new release)

Multi-Agent Code Review (4 agents: Security, Quality, Gemini, Cursor)

Architecture: The three-layer pipeline (agent → schema validation → post-script) is well-designed and ADR 0022 is well-reasoned. The separation of concerns is clean and the zero-trust validation model is sound.

Critical: Post-script may run on unvalidated output

The post-script is registered as a defer in run.go and executes regardless of whether schema validation passed. When validationPassed == false, the function returns an error, but the deferred post-script fires first — meaning post-triage.sh reads the invalid triage-result.json and attempts GitHub mutations with malformed data.

This directly violates ADR 0022: "No unvalidated output is emitted. The pipeline receives a failure signal, not silently bad data."

Fix: Either move post-script execution out of defer (only run after validationPassed == true), or add a guard inside the defer closure.

Inline comments below cover the remaining findings from all 4 review agents, deduplicated and prioritized.

Full review documents:

Research/fullsend-pr279-triage-agent-security-review.md
Research/fullsend-pr279-triage-agent-quality-review.md
Research/fullsend-pr279-triage-agent-gemini-review.md
Research/fullsend-pr279-triage-agent-cursor-review.md

- Fix ADR heading 0020→0022 and stale references in schema/scripts - Add additionalProperties: false to schema root and triage_summary per ADR 0022 zero-trust model - Guard post-script defer on validationPassed so it never runs on unvalidated agent output (critical security fix) - Prefer triage-result.json by name in validate-output-schema.sh instead of non-deterministic find ordering - Update post-triage-test.sh assertions to match labels API refactor - Add GOOS=linux GOARCH=amd64 to vendorBinaryForE2E for cross-compile Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…ell safety - Post-script failure is now a hard error (propagates via named return) - Post-script defer guards on both validationPassed and runErr - Post-script requires validation_loop at harness load time (ADR 0022) - ValidateRunnerEnv accepts an expander function, removing os.Setenv - Schema: maxLength 16384 on comment field - Post-triage.sh: --body-file for shell safety, add_label fails hard - Post-triage.sh: self-duplicate guard (duplicate_of != issue number) - Pre-triage.sh: verify label state after deletion with GET - Shim workflow: concurrency group per issue (cancel-in-progress) - Agent prompt: comment content rules (no @mentions, no verbatim echo) - Agent prompt: proposed_test_case → conceptual description - VendorBinary: file size limit and path validation - Verbose flag comment about artifact upload visibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…agent The load-time check "post_script requires validation_loop" rejected harness/code.yaml which legitimately uses post_script without a validation_loop. The code agent's post-script handles its own output checks and doesn't need a validation loop. - Remove strict Validate() check; move the guard to runtime in run.go where it only skips post-script when validation_loop exists AND failed - Add TestHarnessesLoadAndValidate that extracts the full scaffold and loads every harness YAML through the same pipeline the runner uses - Add GOTOOLCHAIN=auto to cross-compilation env for version portability Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The upload-artifact step was skipped when the fullsend run step failed, losing the transcript needed to debug agent crashes. Adding always() ensures artifacts are uploaded regardless of the run outcome. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

waynesun09

Multi-Agent Review Round 5 — APPROVED

Four review agents (Security, Quality, Gemini, Cursor) all approve after verifying the hardening commit 4fc3690d and 5 supporting commits.

Hardening changes verified (13/13)

Post-script hard error — named return propagates via defer; agent error preserved when both fail
ValidateRunnerEnv expander — os.Setenv eliminated; custom expander function passed directly
Schema maxLength 16384 on comment field — bounds LLM output
--body-file - (stdin piping) — no temp file, no shell expansion risk
add_label fails hard — || true removed; label API failures propagate
Self-duplicate guard — duplicate_of != issue_number with test coverage
Pre-triage label verification — GET after DELETE confirms removal
Concurrency group — triage-${{ issue.number }} with cancel-in-progress
Agent prompt content rules — no @mentions, no verbatim user input echo, no URL relay
proposed_test_case → conceptual — better for heterogeneous repos
VendorBinary — 100MB size limit + directory check
Official sandbox image in triage harness
--verbose flag for claude --output-format stream-json

All prior fixes confirmed intact

Post-script validationPassed guard, ADR 0022 heading, additionalProperties: false, agent-result.json by name, jsonschema >= 4.18.0, category enum, labels API test assertions, GOOS=linux, issue-author re-triage, Makefile targets — all holding.

Remaining follow-ups (non-blocking)

Bot-loop HTML marker (accepted — 3 defense layers exist)
http.DefaultClient in E2E label check (test-only)
Comment content sanitization (deferred to Story 6, #129)

The code policy enumerated specific googleapis.com subdomains but missed the regionless aiplatform.googleapis.com that Claude CLI connects to, causing a 403 policy_denied error. Use *.googleapis.com wildcard to match the triage policy and avoid future breakage from endpoint changes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Fix ADR heading 0020→0022 and stale references in schema/scripts - Add additionalProperties: false to schema root and triage_summary per ADR 0022 zero-trust model - Guard post-script defer on validationPassed so it never runs on unvalidated agent output (critical security fix) - Prefer triage-result.json by name in validate-output-schema.sh instead of non-deterministic find ordering - Update post-triage-test.sh assertions to match labels API refactor - Add GOOS=linux GOARCH=amd64 to vendorBinaryForE2E for cross-compile Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

github-actions Bot deployed to site-preview April 20, 2026 19:35 View deployment

github-actions Bot deployed to site-preview April 20, 2026 20:53 View deployment

github-actions Bot deployed to site-preview April 20, 2026 20:59 View deployment

github-actions Bot deployed to site-preview April 20, 2026 21:09 View deployment

github-actions Bot deployed to site-preview April 20, 2026 23:39 View deployment

github-actions Bot deployed to site-preview April 20, 2026 23:46 View deployment

github-actions Bot deployed to site-preview April 21, 2026 00:32 View deployment

github-actions Bot deployed to site-preview April 21, 2026 00:56 View deployment

github-actions Bot deployed to site-preview April 21, 2026 01:31 View deployment

github-actions Bot deployed to site-preview April 21, 2026 01:50 View deployment

waynesun09 reviewed Apr 21, 2026

View reviewed changes

ralphbean force-pushed the feature/triage-agent-impl branch from d0f1eb9 to c425681 Compare April 21, 2026 14:01

github-actions Bot deployed to site-preview April 21, 2026 14:02 View deployment

github-actions Bot deployed to site-preview April 21, 2026 14:10 View deployment

github-actions Bot deployed to site-preview April 21, 2026 14:11 View deployment

ralphbean force-pushed the feature/triage-agent-impl branch from 4bc608d to 1b26cc8 Compare April 21, 2026 14:11

github-actions Bot deployed to site-preview April 21, 2026 14:12 View deployment

github-actions Bot deployed to site-preview April 21, 2026 14:35 View deployment

github-actions Bot deployed to site-preview April 21, 2026 14:49 View deployment

github-actions Bot deployed to site-preview April 21, 2026 14:51 View deployment

waynesun09 mentioned this pull request Apr 21, 2026

feat(agents): add review agent and pr-review skill #171

Merged

4 tasks

ralphbean force-pushed the feature/triage-agent-impl branch from 49b77b6 to 7c8532a Compare April 21, 2026 15:53

github-actions Bot deployed to site-preview April 21, 2026 15:54 View deployment

github-actions Bot deployed to site-preview April 21, 2026 19:33 View deployment

github-actions Bot deployed to site-preview April 21, 2026 19:58 View deployment

github-actions Bot deployed to site-preview April 21, 2026 20:05 View deployment

waynesun09 approved these changes Apr 21, 2026

View reviewed changes

github-actions Bot deployed to site-preview April 21, 2026 20:19 View deployment

ralphbean added this pull request to the merge queue Apr 21, 2026

Merged via the queue into main with commit 6a9821a Apr 21, 2026
5 checks passed

ralphbean deleted the feature/triage-agent-impl branch April 21, 2026 20:38

ralphbean mentioned this pull request Apr 21, 2026

feat: add --vendor-fullsend-binary for dev iteration #277

Closed

4 tasks

ralphbean mentioned this pull request Apr 29, 2026

Add the triage skills and strategies experiments fullsend-ai/experiments#12

Closed

fullsend-ai-retro Bot mentioned this pull request Jul 7, 2026

Evidence for #1574: 20 stale ready-for-merge labels on open PRs in repo with auto_merge disabled #3305

Closed

guyoron1 mentioned this pull request Jul 7, 2026

Evidence for #1574: 20 stale ready-for-merge labels on open PRs in repo with auto_merge disabled guyoron1/fullsend#129

Closed

fullsend-ai-retro Bot mentioned this pull request Jul 7, 2026

Evidence for #3317 and #3312: consecutive retro on trivial bot dep bump hours after identical retro (segment-bridge #279) #3341

Closed

qodo-code-review Bot mentioned this pull request Jul 23, 2026

fix(#5188): re-trigger review via label after fix-agent push #5551

Closed

2 tasks

qodo-code-review Bot mentioned this pull request Sep 7, 2026

fix(#827): dismiss stale bot approvals #7075

Merged

8 tasks

Navigation Menu

Uh oh!

feat: implement triage agent (Story 3, #126) - #279

feat: implement triage agent (Story 3, #126)#279
ralphbean merged 31 commits into
mainfrom
feature/triage-agent-impl

ralphbean commented Apr 20, 2026

Uh oh!

github-actions Bot commented Apr 20, 2026 •

edited

Loading

Uh oh!

waynesun09 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waynesun09 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ralphbean commented Apr 20, 2026

Summary

Key changes

Also includes (cherry-picked from #277)

Test plan

Related

Uh oh!

github-actions Bot commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Site preview

Uh oh!

waynesun09 left a comment

Choose a reason for hiding this comment

Multi-Agent Code Review (4 agents: Security, Quality, Gemini, Cursor)

Critical: Post-script may run on unvalidated output

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waynesun09 left a comment

Choose a reason for hiding this comment

Multi-Agent Review Round 5 — APPROVED

Hardening changes verified (13/13)

All prior fixes confirmed intact

Remaining follow-ups (non-blocking)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Apr 20, 2026 •

edited

Loading