Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
eea4934
docs: add ADR for harness-level output schema enforcement
maruiz93 Apr 16, 2026
dd372c5
feat(triage): implement triage agent with structured JSON output (Sto…
ralphbean Apr 20, 2026
c8072f0
fix: address review findings — schema correctness, URL validation, en…
ralphbean Apr 20, 2026
1c18260
feat: add --vendor-fullsend-binary flag for dev iteration
ralphbean Apr 20, 2026
7a12efc
fix: FULLSEND_DIR validation ordering and script execute permissions
ralphbean Apr 20, 2026
d75de8e
fix: renumber duplicate ADR and vendor binary for e2e triage test
ralphbean Apr 20, 2026
bdbc987
fix: set module root as working dir for e2e binary build
ralphbean Apr 20, 2026
ffc15ab
fix: log first 4000 chars of failed workflow run for debugging
ralphbean Apr 20, 2026
0aea013
fix: improve workflow failure diagnostics with step-level status
ralphbean Apr 20, 2026
480888b
fix: skip triage smoke test when inference credentials unavailable
ralphbean Apr 20, 2026
4e853a6
fix: extract enrollment PR merge so unenrollment works without triage…
ralphbean Apr 21, 2026
5ea6476
fix: remove ready-to-triage label after triage completes
ralphbean Apr 21, 2026
47299a2
fix: use labels API to prevent triage re-dispatch loop
ralphbean Apr 21, 2026
4f39cd7
refactor: remove ready-to-triage label management
ralphbean Apr 21, 2026
29d3e61
fix: address PR #279 review findings
ralphbean Apr 21, 2026
dece0ca
fix: add jsonschema to sandbox image
ralphbean Apr 21, 2026
25fe16e
fix: review follow-ups — category enum, no glob fallback, jsonschema …
ralphbean Apr 21, 2026
ecb95cb
fix: review round 2 — generic output filename, category sync, dead co…
ralphbean Apr 21, 2026
ad0a146
feat: add make script-test and make test targets
ralphbean Apr 21, 2026
3e68884
chore: remove stale triage agent plan document
ralphbean Apr 21, 2026
27f57c7
fix: make post-script failure a hard error
ralphbean Apr 21, 2026
b1eff43
fix: allow issue authors to re-trigger triage regardless of association
ralphbean Apr 21, 2026
6992c9f
fix: create .claude/hooks directory before copying security hooks
ralphbean Apr 21, 2026
76b34c1
fix: add --verbose flag required by claude --output-format stream-json
ralphbean Apr 21, 2026
29f3e3c
fix: bump sandbox creation timeouts to avoid transient failures
ralphbean Apr 21, 2026
6e1e321
Revert "fix: bump sandbox creation timeouts to avoid transient failures"
ralphbean Apr 21, 2026
f315448
fix: point triage harness at official sandbox image
ralphbean Apr 21, 2026
4fc3690
fix: address review feedback — harden post-script, validation, and sh…
ralphbean Apr 21, 2026
1490af9
fix: remove strict post_script/validation_loop check that broke code …
ralphbean Apr 21, 2026
2fef95e
fix: upload artifacts on failure so transcripts are always available
ralphbean Apr 21, 2026
4c4a45a
fix: use wildcard for googleapis.com in code agent network policy
ralphbean Apr 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
.DEFAULT_GOAL := help
.PHONY: help bootstrap lint lint-all check fmt \
mindmap go-build go-test go-lint go-fmt go-vet go-tidy e2e-test e2e-playwright \
e2e-export-session e2e-upload-session
mindmap go-build go-test go-lint go-fmt go-vet go-tidy \
script-test test \
e2e-test e2e-playwright e2e-export-session e2e-upload-session

help:
@echo "Available targets:"
Expand All @@ -18,6 +19,8 @@ help:
@echo " go-fmt - Format Go code"
@echo " go-vet - Run go vet"
@echo " go-tidy - Run go mod tidy"
@echo " script-test - Run shell script tests (post-triage, validate-output-schema)"
@echo " test - Run all checks: lint, go-vet, go-test, script-test"
@echo " e2e-test - Run admin e2e tests (requires E2E_GITHUB_SESSION_FILE or E2E_GITHUB_USERNAME + E2E_GITHUB_PASSWORD)"
@echo " e2e-export-session - Login to GitHub and export a Playwright session file"
@echo " e2e-upload-session - Export session and upload it as a GitHub repo secret"
Expand Down Expand Up @@ -89,6 +92,12 @@ go-vet:
go-tidy:
go mod tidy

script-test:
bash internal/scaffold/fullsend-repo/scripts/post-triage-test.sh
bash internal/scaffold/fullsend-repo/scripts/validate-output-schema-test.sh

test: lint go-vet go-test script-test

E2E_SESSION_FILE ?= $(CURDIR)/.playwright/session.json

e2e-test: e2e-playwright
Expand Down
134 changes: 134 additions & 0 deletions docs/ADRs/0022-harness-level-output-schema-enforcement.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
title: "22. Harness-level output schema enforcement"
status: Accepted
relates_to:
- security-threat-model
- agent-architecture
topics:
- security
- output-validation
- harness
---

# 22. Harness-level output schema enforcement

Date: 2026-04-15

## Status

Accepted

## Context

[ADR 0018](0018-scripted-pipeline-for-multi-agent-orchestration.md) requires
that agent outputs follow a structured contract so pipelines can evaluate
conditions and pass data between stages. It does not say where that contract is
enforced. [ADR 0016](0016-unidirectional-control-flow.md) establishes that
control flows strictly downward through the execution stack and that the
harness defines what the runtime can do — the runtime cannot modify its own
harness. [ADR 0017](0017-credential-isolation-for-sandboxed-agents.md)
establishes that agents run inside sandboxes with restricted networking, with
host-side pre-scripts and post-scripts — both part of the harness — handling
data prefetch and output application. This post-script pattern already
processes agent output in a controlled, deterministic environment on the host.

The [threat model](../problems/security-threat-model.md) identifies
agent-to-agent prompt injection (Threat 5) as a distinct risk: a compromised
agent's output is consumed by downstream agents. Zero trust between agents
means every agent's output must be validated regardless of source. This
decision addresses where and how that structural validation occurs.

## Options

### Option A: Harness post-script enforcement

A harness post-script validates agent output against a declared schema on the
host, after the runtime finishes and before output reaches the pipeline or
downstream consumers. The schema is part of harness configuration — immutable
from the runtime (ADR 0016), governed by CODEOWNERS. Non-compliant output
triggers a retry: the harness re-invokes the runtime with the schema violation
fed back. Retries are capped; exhaustion is a hard failure.

**Trade-offs:** Runs on the host in a controlled environment — the runtime
cannot bypass it. Every agent is validated, not just pipeline endpoints.
Retries cost time and money.

### Option B: Pipeline-level enforcement

The pipeline executor validates agent output between stages, after it has left
the sandbox.

**Trade-offs:** Simpler to implement — one validation point per pipeline. But
single-agent invocations (no pipeline) skip validation entirely. Malformed
output has already exited the sandbox before validation occurs, widening the
window for downstream consumption of bad data. Does not satisfy zero trust:
the pipeline must be aware of every agent's schema.

### Option C: Dedicated validator agent

A separate LLM-based agent checks each agent's output for correctness and
safety.

**Trade-offs:** Can perform semantic checks beyond structural validation. But
adds latency, cost, and a new attack surface — the validator itself can be
compromised or manipulated. Non-deterministic: the validator may disagree with
itself across runs. Structural validation does not require an LLM.

### Option D: No enforcement (trust agent output)

Agents are expected to produce correct output via prompt engineering alone.

**Trade-offs:** Zero overhead. But prompt engineering cannot guarantee output
structure — LLMs produce non-compliant output unpredictably. Violates zero
trust. A compromised agent's malformed output propagates silently.

## Decision

The harness validates every agent's output against a declared JSON schema
via a post-script on the host, after the runtime finishes and before the
output reaches the pipeline or downstream consumers. The mechanism:

1. The harness configuration declares an output schema for the agent.
2. After the runtime produces output, a harness post-script on the host
validates it against the schema.
3. If validation fails, the harness feeds the violation back to the runtime
and re-invokes it, up to a configured retry limit.
4. If retries are exhausted, the agent fails. No unvalidated output is
emitted. The pipeline receives a failure signal, not silently bad data.

Schema definitions are part of the harness configuration — governed by
CODEOWNERS and immutable from the runtime per ADR 0016. Specific per-agent
schemas are deferred to normative specs
([ADR 0015](0015-normative-specifications-directory.md)).

This extends the post-script pattern established in ADR 0017: where ADR 0017
uses post-scripts for application-level actions (applying labels, posting
comments), this ADR adds structural schema validation as a prior step. Schema
validation gates the output before any application-level post-scripts consume
it.

## Consequences

- **Schema validation is a security layer, not the security layer.** It
catches crude compromises (wrong format, missing fields) but not
sophisticated ones (conformant structure, malicious content). Content-level
sanitization of string fields — including Unicode injection payloads in
structured output — is a separate concern. This is one layer in a
defense-in-depth model.
- **Retry exhaustion is a hard failure.** The system never falls back to
emitting unvalidated output. This trades availability for integrity —
acceptable in a zero-trust model where silent bad data is worse than a
visible failure.
- **Schema and prompt must be versioned together.** If the schema changes but
the agent's prompt still describes the old format, the agent will fail
validation on every attempt. Both artifacts live in the harness
configuration and should be updated atomically.
- **Retries have a cost.** Each retry is a full LLM invocation. The retry
budget is a trade-off between resilience (more retries tolerate transient
non-compliance) and cost (each retry costs time and money). The budget
should be low — 1-2 retries — because a well-prompted agent with a clear
schema should comply on the first attempt; repeated failure suggests a
deeper problem that more retries will not fix.
- **Every agent is validated, not just pipeline endpoints.** In a multi-agent
pipeline where parallel agents feed into an aggregator, each agent's output
is schema-checked independently before the aggregator sees it.
8 changes: 8 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,14 @@ The harness is what makes a generic LLM into a specific agent with a specific ro

The harness draws its configuration from the adopting organization's **`.fullsend`** repository — skills, workflow definitions, and agent behavioral instructions are assembled from the layered config (fullsend defaults, then org config, then per-repo overrides). (See [ADR 0003](ADRs/0003-org-config-repo-convention.md).)

**Decided:**

- Output schema enforcement: a harness post-script validates every agent's
output against a declared JSON schema on the host. Non-compliant output
triggers a retry (capped); exhaustion is a hard failure — no unvalidated
output is emitted
([ADR 0022](ADRs/0022-harness-level-output-schema-enforcement.md)).

**Open questions:**

- Does the harness live inside the sandbox (configuring the agent from within its isolation boundary) or outside it (preparing the environment before the agent starts)?
Expand Down
Loading
Loading