chore(tools): execute the harness in CI instead of only reading it

[thomasluizon]

The hole

The repo already has reviewer independence: claude-review.yml runs a fresh Claude session on every PR. What it does not do is execute the harness. It reviews the diff. So a broken tool can be read, approved and merged.

node .claude/hooks/test-hooks.mjs was the only executable proof in this layer, and it covers hooks only. Everything under tools/ (16 scripts) had no executable gate at all.

The measured evidence

• One session built a harness, passed its own checks, and an independent session found 7 Critical/High defects in the same commit range. Four of the seven were findable only by running the documented sequence end to end, not by reading the diff.
• tools/launch-worker.mjs shipped in PR chore(orchestrate): make the six measured worker-launch gotchas structural #604 treating orca terminal wait's "not yet" (exit 1 carrying an ok:false timeout payload) as a fatal error, which would have broken the exact loop the tool exists for. Reading it did not catch that; running it did.

What the gate executes

node tools/test-tools.mjs, wired into guards.yml as the Harness Execution job (which also runs node .claude/hooks/test-hooks.mjs), scoped to tools/** and .claude/** by a detect step in the Expo SDK Pin job's style, so the job always reports and can be made a required check.

Three layers:

1. Structural coverage. Enumerates tools/*.{mjs,sh,ps1} and fails on any script with no coverage entry. This is the part that keeps the suite from rotting at tool N+1.
2. The universal contract (tools/CONVENTIONS.md), per script: --help exits 0 with usage on stdout, and invalid input is refused before the tool does any work. .sh runs under a resolved real bash (the PATH bash on Windows is the WSL stub), .ps1 under pwsh.
3. Decision paths, hermetic. orca is stubbed by pointing ORCA_BIN at the node binary with a preloaded shim that answers a per-case plan; an unstubbed orca call exits 9 with a stub-miss payload, so an unexpected external call is a loud failure rather than a silent pass. No worktree is created, no network is touched, no Linear issue is read. Covered today: wave-plan (blockedBy DAG ordering, launchable set, empty project), check-dashes, check-ticket, check-push-target, check-frontmatter (cwd safety), plus the three orchestrate tools below.

The three orchestrate tools

launch-worker.mjs, nudge-worker.mjs and worker-status.mjs live on the open PR #604 branch, not on main, so on this branch they print PENDING and their cases are inert. They were written and verified against that branch's head by materialising the three files locally (uncommitted) and running the suite: all 18 cases pass. The moment #604 merges, they execute for real. Cases include repo resolution from a repo:* label, refusal when the label has no repos entry, refusal of a prompt file inside a repo, refusal of a non-interactive engine (interactive: false and the field omitted entirely), refusal of headless args behind an interactive: true declaration, worker:sonnet model routing, rejection of multi-line nudge text, and the exit-code-vs-payload distinction on orca terminal wait (timeout payload means busy: exit 1, nothing sent; any other ok:false means tool error: exit 3).

Red proof 1: break a tool, the gate goes red

Reintroduced PR #604's original defect in nudge-worker.mjs (if (result.status !== 0) fail(3, ...) before parsing the payload):

FAIL nudge-worker.mjs: refuses to send while the worker is busy
     nudge-worker.mjs --terminal t1 --text hi --wait-attempts 1
     exit 3, expected 1
     orca terminal wait failed with exit 1
FAIL nudge-worker.mjs: an orca failure that is not a timeout is a tool error
     nudge-worker.mjs --terminal t1 --text hi --wait-attempts 1
     stderr did not match /unknown handle/
     orca terminal wait failed with exit 1

ORBIT TOOLS GATE FAILED (2)
EXIT=1

Reverted; the suite is green again.

Red proof 2: add an uncovered tool, the structural guard goes red

Added a dummy tools/zz-probe.mjs with no coverage entry:

# structural coverage
FAIL every tools/ script has coverage (20 scripts)
     no COVERAGE entry for: zz-probe.mjs
     Add one to INVALID_INPUT (and a gateCases entry if it has decision paths) in tools/test-tools.mjs.
FAIL zz-probe.mjs: --help exits 0 with usage on stdout
     zz-probe.mjs --help
     stdout did not match /usage|Usage/
     probe

ORBIT TOOLS GATE FAILED (2)

Probe deleted.

What the gate found on main (fixed here)

The contract in tools/CONVENTIONS.md was prose, so seven of sixteen scripts did not meet it:

Tool	Defect	Fix
check-copy, check-dashes	--help exited 2	--help prints usage, exits 0
check-frontmatter, check-suppressions-ratchet, redesign-coverage	--help ran the tool instead of printing usage	usage block plus unknown-argument rejection
check-push-target	no --help, silent exit 0	usage block, arguments rejected
arch-map, surface-manifest	a bogus flag silently rewrote architecture.json / surfaces.json and exited 0	reject unknown arguments with exit 2, before any write
merge-sweep, merge-sweep-cov	a bogus repo slug or PR number reached the GitHub API as a 404 that printed SKIP and exited 0, so a typo read as "nothing to merge"	validate <owner/repo> and numeric PR ids before the first gh call, exit 2
check-ticket --file <missing>	unhandled ENOENT stack trace	exit 2 with the usage error
check-frontmatter	resolved .claude/skills from the caller's cwd, so it failed from any subdirectory	resolve from the script's own location (cwd-safe, per the contract)

Two tools were also missing from the tools/README.md catalog (check-frontmatter, check-push-target); both are added.

The judgement half

Review rubric dimension 15: a diff touching tools/**, .claude/skills/**, .claude/agents/**, .claude/hooks/** or .claude/orchestrator.json requires executed evidence. The reviewer cites the Harness Execution result; a red job is Critical, a job that never ran on an in-scope diff is High, a new tool with no coverage is High, and a claim of "verified" with no command behind it is itself a High finding. Kept consistent with the Signal gate: it fires on missing execution and missing coverage, never on a tool's style or flag naming.

Docs

tools/README.md (catalog rows, At a glance), tools/CONVENTIONS.md (new "The gate" section, At a glance), TESTING.md (ninth suite, CI mapping; the hook suite is no longer local-only), root CLAUDE.md (Code standards gate list, a Conventions bullet, the hooks bullet).

Verification

• node tools/test-tools.mjs green on this branch (16 scripts, 3 pending) and green with the three PR chore(orchestrate): make the six measured worker-launch gotchas structural #604 tools materialised (19 scripts, 0 pending).
• node .claude/hooks/test-hooks.mjs, node tools/check-frontmatter.mjs, node tools/check-copy.mjs --check, node tools/check-dashes.mjs --files <changed>, npm run lint, npm run type-check: all green.
• Normal paths of every patched tool re-run by hand (surface-manifest --baseline origin/main, redesign-coverage --json, check-suppressions-ratchet, capture-surfaces --filter, merge-sweep <valid slug> zzz).

Deliberately out of scope

worker-status.mjs gets contract coverage only. Every one of its checks reads a real git worktree, gh and Linear, and stubbing gh needs an executable shim, which Node refuses to spawn from a .cmd on Windows. A hermetic case for it needs a git-fixture harness that is its own change.

Not for merge yet.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LBFbK416nnXc7kJwkknZpo

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
orbit-ui-mobile-web	Ignored		Jul 25, 2026 1:58am

[claude]

Code Review: PR #606

Recommendation: APPROVE

Summary

PR #606 adds tools/test-tools.mjs, wired into guards.yml as the "Harness Execution" job, which actually runs every script in tools/ against the CONVENTIONS.md CLI contract (--help exits 0, bad input refused before any work) plus per-tool decision paths, with orca stubbed hermetically via a --require-preloaded shim on NODE_OPTIONS. It fixes seven real contract violations the new gate found on main (broken/missing --help, silently-swallowed bad arguments, a cwd-unsafe check-frontmatter.mjs), adds rubric dimension 15 ("harness changes need EXECUTED evidence"), and updates TESTING.md, tools/CONVENTIONS.md, tools/README.md, and CLAUDE.md in the same PR. Scope is entirely tools/, .claude/, .github/workflows/guards.yml, and root docs — no apps/*, packages/shared, or orbit-api surface touched, so five of six subagents and six rubric dimensions are cleanly N/A.

Static verification: traced the orca-stub mechanism (correct — the shim no-ops when argv[1] resolves to a real file on disk, i.e. when node is running the tool itself, and only answers the stub plan when it's being invoked as orca's subcommand path), confirmed all 13 patched scripts' new --help/unknown-argument logic is internally consistent with their existing usage/exit-code conventions, verified merge-sweep-cov.sh defines the usage() function the new validation block calls, and cross-checked the PR body's "16 scripts, 3 pending" claim against the actual tools/ directory listing — it matches exactly (launch-worker.mjs, nudge-worker.mjs, worker-status.mjs are genuinely absent from this branch, consistent with living on the still-open PR #604).

Findings

Critical: None
High: None
Medium: None
Low/Info:

• [Info] tools/test-tools.mjs is well-structured: no function near the 50-line soft cap, and the orca stub is a genuine root-cause solution for hermetically testing tools that shell out to orca, not a workaround (CLAUDE.md rules 3, 5).
• [Info] tools/capture-surfaces.mjs:40 — the new --api-base line in the printed USAGE block documents a flag whose parsing and four call sites already existed pre-PR; confirmed via grep so it isn't miscounted as new, ungated functionality.

Subagents

All five (parity-checker, i18n-syncer, contract-aligner, security-reviewer, design-reviewer) — N/A, none of their gating surfaces changed.

Validation

Per the CI adaptation for this PR: Build, Unit Tests, and SonarCloud are separate required checks (skipped here). All relevant guards.yml jobs are green on the head commit (e5178a3a): Harness Execution, Lint, Type Check, Dash Ban, Copy Register, Suppressions Ratchet, Expo SDK Pin, Cross-Platform Parity, Contract Drift, Design Token Guard, Skill and Agent Frontmatter, Dependency Audit, Architecture map drift — all SUCCESS.

Deferred

• Dimensions 8, 9, 10, 11, 13, 14 — N/A, none of their surfaces touched.
• Cross-repo dimensions (contract-aligner's orbit-api side, the orbit-api half of the backward-compat guard) — not verifiable in this job (orbit-api not checked out).
• All changed files received an explicit verdict; nothing else deferred.

What's good

Genuine root-cause fix matching its own thesis (runs code instead of reading it), with two "red proof" demonstrations in the PR body showing the gate actually fails on a reintroduced defect and an uncovered new script. Every patched tool got the same minimal, symmetric treatment. check-frontmatter.mjs's cwd-unsafety fix is a real fix, not a patch. worker-status.mjs is honestly scoped out with a stated reason rather than faked. Docs updated in the same PR, including an incidental fix to a pre-existing CLAUDE.md gate-list gap.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[claude]

Code Review: PR #606

Recommendation: APPROVE

Summary

Adds tools/test-tools.mjs, a hermetic harness-execution gate that runs every script in tools/ against the CONVENTIONS.md CLI contract (--help exits 0, invalid input refused before work) plus per-tool decision paths, with orca stubbed via a NODE_OPTIONS --require shim. Wires it into a new Harness Execution job in guards.yml (path-scoped to tools/** / .claude/**), fixes the seven contract violations the gate found on main, and adds rubric dimension 15 (harness changes need EXECUTED evidence) plus matching doc updates.

I traced every new/changed script against its test-tools.mjs case by hand (argument parsing, exit codes, stderr patterns) — launch-worker.mjs's 11 decision-path cases, nudge-worker.mjs's 7, wave-plan.mjs's DAG-ordering cases, and the --help/invalid-input contract on all 19 covered scripts — and found no mismatch between the stubbed expectation and the real implementation. The PR's own motivating example (orca terminal wait's exit-1 timeout payload) is correctly modeled in both launch-worker.mjs and nudge-worker.mjs, and the matching test-tools.mjs cases (BUSY_STUB, BROKEN_STUB) exercise the exit-code-vs-payload branch, not just the exit code.

Findings

Critical: None. High: None. Medium: None.

Info — gateCases["check-frontmatter.mjs"]'s "runs from any cwd" case reads the repo's actual .claude/skills/.claude/agents tree rather than a staged fixture: hermetic per CONVENTIONS.md (no network/worktree/Linear/writes outside the temp root) but not fixture-isolated, so an unrelated frontmatter regression elsewhere would fail this case for a reason unrelated to cwd-independence. Reasonable trade-off given check-frontmatter.mjs already has its own CI job; no action needed.

Subagents

parity-checker / i18n-syncer / contract-aligner / security-reviewer / design-reviewer: all N/A — the diff touches only tools/**, .claude/**, .github/workflows/guards.yml, CLAUDE.md, TESTING.md; no apps/* or orbit-api file changed.

Validation

Dimension 15 (Harness changes need EXECUTED evidence) requires citing the Harness Execution job's result for this PR: PASS — confirmed via gh pr view 606 --json statusCheckRollup, Harness Execution: SUCCESS. Full rollup for the head SHA is green across the board (Lint, Type Check, Unit Tests, Build, Dash Ban, Copy Register, Suppressions Ratchet, Expo SDK Pin, Cross-Platform Parity, Contract Drift, Architecture map drift, Skill and Agent Frontmatter, SonarCloud, CodeQL, GitGuardian, Dependency Audit/Review).

Deferred

Dimensions 8/9/10/12(frontend)/13/14: N/A, no apps/*/orbit-api surface touched. Dimension 11 (contract drift): N/A, no Zod schema or DTO field touched.

What's good

• Genuinely hermetic gate: orca stubbed via shim that exits 9 (loud failure) on any unstubbed call; no worktree, no network, no Linear touched.
• Structural coverage closes exactly the named hole (tool N+1 cannot land uncovered); verified against the current 19-script tools/ directory — all covered.
• The two "red proof" demonstrations in the PR body (reintroducing the nudge-worker.mjs defect, adding an uncovered zz-probe.mjs) are backed by real, checkable code paths, not just prose.
• Docs (TESTING.md, tools/README.md, tools/CONVENTIONS.md, CLAUDE.md) stay accurate to the CI wiring they describe.

Recommendation

Approve as-is. PR #604 (the three orchestrate tools this PR adds coverage for) is already merged to main, so the PR body's "PENDING" caveat no longer applies at merge time — this branch's own CI run shows the full 19-script suite green.