chore(harness): enforce read-only audit agents at the tool layer (gate over prose) - #449
Conversation
…e over prose)
The prod-readiness and audit workflows are read-only by contract ("assess, never
edit"), but their Haiku finders / critics / skeptics / ops agents were spawned with
the default write-capable tool profile. On 2026-07-10 a prod-readiness run drifted
from describing fixes to applying them, writing ~20 stray files across both repos
(an invented feature, broken test edits, a dependency install). The contract lived
only in prose, so nothing stopped it.
Make it a gate (root CLAUDE.md rule 6):
- New audit-readonly subagent (tools: Glob, Grep, Read only; no Write/Edit/Bash),
plus its .opencode pointer (edit/bash/task/webfetch/websearch denied).
- audit.mjs: every finder / skeptic / critic / gap-finder agent() now runs as
agentType: 'audit-readonly'.
- prod-readiness.mjs: the ops + verify agents likewise.
- Soften the tests-audit prompt ("SPECIFY the missing test as text in the fix
field, never write it to disk") as defense-in-depth behind the tool gate.
Takes effect next session (the agent registry loads at startup). Refs #243
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review Complete
Scope: PR #449 in thomasluizon/orbit-ui-mobile — chore(harness): enforce read-only audit agents at the tool layer
Recommendation: APPROVE
| Severity | Count |
|---|---|
| Critical (incl. old-client breaks) | 0 |
| High | 0 |
| Medium | 2 |
| Low / Info | 0 |
Code Review: PR #449
Scope: PR #449, thomasluizon/orbit-ui-mobile — 4 files changed: .claude/agents/audit-readonly.md (new), .claude/workflows/audit.mjs (5 lines), .claude/workflows/prod-readiness.mjs (2 lines), .opencode/agents/audit-readonly.md (new).
Summary
The PR closes a real incident (2026-07-10: unrestricted Haiku finder/skeptic/critic agents drifted from describing fixes to writing ~20 stray files) by moving the read-only contract from prose into a tool-layer gate. It adds a new audit-readonly subagent restricted to tools: Glob, Grep, Read and wires agentType: 'audit-readonly' into every finder/skeptic/critic/gap-finder call in audit.mjs and every ops/verify call in prod-readiness.mjs, plus the matching .opencode pointer. The new agent files match the established convention exactly (checked against contract-aligner.md/security-reviewer.md and their pointers). No apps/*, packages/shared, or orbit-api files are touched.
Findings
Critical / High
None.
Medium
[Medium] Prompt-softening is inconsistent — the shared base template keeps the exact phrase implicated in the incident
· dimension: 1 Correctness / 5 No-workaround (defense-in-depth layer)
· location: .claude/workflows/audit.mjs:157 (finderPrompt, shared by all four kinds)
· issue: The PR body itself quotes "fix (the concrete change)" as one of the two phrasings that drifted agents from describing to applying fixes. That exact string is untouched in the shared finderPrompt template used by every kind. Only the tests-kind extra field (line 92) got reworded to add "as text in the fix field (never write it to disk)".
· risk: If the tool-layer gate ever fails to engage for a kind (see next finding), security/performance/code-quality finders carry the original ambiguous instruction with none of the defense-in-depth wording tests got.
· fix: Apply the same softening to line 157, e.g. fix (the concrete change, described as text — never written to disk).
· reference: PR body "Why" section; CLAUDE.md rule 1.
[Medium] No tracked step confirms agentType: 'audit-readonly' is actually honored by the harness
· dimension: 1 Correctness
· location: .claude/workflows/audit.mjs:223,237,262,271; .claude/workflows/prod-readiness.mjs:112,129
· issue: agentType has zero prior occurrences anywhere in this repo's history (verified via git log --all -S"agentType") — it's a brand-new option key on a host-provided agent() function with no local definition to confirm it's recognized. The PR body honestly admits it "can't be smoke-tested in the session that authored it," but its own follow-up list only proposes a /lesson capture, not a check that the gate engaged.
· risk: If the harness silently ignores an unrecognized options key, this merges reading as fixed while enforcing nothing — the same silent-miss failure mode, one layer up.
· fix: Add a concrete next-session step: confirm via workflow logs (or a deliberate "try to write a file" canary) that a spawned finder actually lacks Write/Edit/Bash, tracked alongside the planned /lesson.
· reference: PR body "Notes / verification"; CLAUDE.md rule 6.
Low / Info
None posted (signal gate).
Subagents
| Agent | Verdict |
|---|---|
| parity-checker | N/A — no apps/web/**/apps/mobile/** changed |
| i18n-syncer | N/A — no user-facing strings/locale JSON changed |
| contract-aligner | N/A — no packages/shared/src/types/*/DTOs changed |
| security-reviewer | N/A — orbit-api untouched |
| design-reviewer | N/A — no UI files changed |
Validation
| Check | Result |
|---|---|
| Lint / Type check / Tests / Build | N/A — diff touches only .claude//.opencode/ harness config, no lintable/testable app surface; PR's own Build/Unit Tests/SonarCloud checks cover this separately |
Deferred
- Dimensions 8, 9, 10, 11, 13, 14: N/A — none of their surfaces (UI, web/mobile, i18n, shared/DTO contracts, orbit-api, user-facing feature surface) are touched by this diff.
- Adversarial skeptic / cross-model second opinion: not run — gated to Critical/High, and none survived here.
- All 4 changed files were reviewed against the committed PR diff — nothing left unreviewed.
What's good
- Textbook gates-over-prose fix — read-only enforced by tool absence, not a trusted prompt.
- New agent file +
.opencodepointer match the existing convention exactly. - Every finder/skeptic/critic/gap-finder call site in both workflow files was updated — none missed.
- Unusually honest "Notes/verification" section that doesn't overclaim the fix is proven.
Recommendation
APPROVE. No Critical/High blocks merge. Two Medium follow-ups worth doing in the next live session: even out the prompt-softening wording across all four audit kinds, and actually confirm agentType: 'audit-readonly' restricts tools in a real run rather than assuming the harness recognizes the new key.
1 participant
Why
The
/audit-*and/prod-readinessworkflows are read-only by contract ("assess, never edit"), but their Haiku finders / critics / skeptics / ops agents were spawned viaagent(...)with noagentType— i.e. the default write-capable tool profile. Nothing at the tool layer enforced the contract.On 2026-07-10 a
/prod-readinessrun (423 agents) drifted from describing fixes to applying them — prompted with "write the CONCRETE missing test" and "fix (the concrete change)" while holding Write/Edit/Bash, the agents wrote ~20 stray files across both repos: an invented avatar-upload feature, new + edited test files (several non-compiling), and a@next/bundle-analyzerinstall. Discovered because they polluted a working tree and broke the build. (The stray files are preserved onsalvage/audit-spew-243in each repo for later triage.)This is a textbook gates-over-prose miss (root
CLAUDE.mdrule 6): a prose contract is advisory and decays; make it a tool-level gate.Fix
audit-readonlysubagent (.claude/agents/audit-readonly.md,tools: Glob, Grep, Readonly — no Write/Edit/Bash) + its.opencodepointer (edit/bash/task/webfetch/websearch denied).audit.mjs: every finder / skeptic / critic / gap-finderagent()now runs asagentType: 'audit-readonly'.prod-readiness.mjs: the ops + verify agents likewise.With the tool gate, prompt drift can no longer touch disk regardless of phrasing.
Notes / verification
.mjsfiles parse in the harness execution context;.gitattributeskeeps them LF; diff is minimal (5 lines inaudit.mjs, 2 inprod-readiness.mjs)./lessonso it graduates into the harness rules.Refs #243.
🤖 Generated with Claude Code