feat(#608): diagnose un-normalized tool names in allowlist hook

[fullsend-ai-coder]

Summary

Runtime adapters must translate their tool names to Claude vocabulary before the sandbox hooks run (ADR 0090); the pi adapter does. When one does not — an extension tool, a future adapter without a map, a pi version that adds a tool — tool_allowlist_pretool.py blocked the call with the same NOT in the allowlist reason as a genuinely forbidden tool, hiding the adapter gap behind a plausible allowlist denial. This PR makes that gap diagnosable without weakening the fail-closed, exact-match allowlist — no case-insensitive allowing.

Closes #608

Changes

• internal/security/hooks/tool_allowlist_pretool.py — a blocked name that equals an allowlisted entry case-insensitively is reported as a normalization gap with an ALLOWLIST_HOOK_ERROR reason that says which side is wrong:
  • allowlist entry is a canonical/legacy Claude name (bash vs Bash, ls vs LS) → "the runtime adapter must translate it", finding tool_name_unnormalized (high, block);
  • tool name is the Claude one but the allowlist is not (Bash vs bash) → "FULLSEND_TOOL_ALLOWLIST entry 'bash' is not canonical … fix the allowlist", finding allowlist_entry_unnormalized (high, block);
  • neither is Claude vocabulary → says so, blames neither, finding tool_name_case_collision.
    MCP names (mcp__*) are matched verbatim and keep the plain tool_blocked (critical) path. A non-string tool_name, or a JSON payload that is not an object ([]/null/123 — pre-existing gap), now blocks with the JSON contract instead of a Python traceback. The three new findings are high, not critical, so an adapter gap does not trip critical-keyed escalation the way a forbidden tool does. Uses casefold(); when several entries fold together a known Claude name wins.
• internal/security/canonical_tools.go — CanonicalClaudeTools lists the tool names Claude Code exposes, verified 2026-08-23 against the live Claude Code tools reference (latest release, 2.1.241 at the time; the CHANGELOG records no tool additions/removals since the 2.1.234 pinned in the sandbox image — re-check on every pin bump); LegacyClaudeTools lists names agents and adapters still use (LS, MultiEdit→Edit, Task→Agent, TodoRead, NotebookRead→Read); KnownClaudeTool(). The hook carries a mirror of both sets (scripts cannot reach Go).
• Tests — TestToolAllowlistHook_VocabularyMatchesGo parses the embedded script and fails on any drift between the Go and Python sets; TestHookPlan_ToolsAreCanonical pins every HookPlan tool (defaults and every hook enabled); pythonNameSet ignores commented-out entries (own fixture test); TestCanonicalClaudeTools_WellFormed (no (/,, no mcp__, canonical ∩ legacy = ∅, legacy replacements canonical); internal/runtime/pi_agent_test.go checks the pi maps in both directions and that every name is canonical-or-legacy Claude vocabulary — a deliberate relaxation of the issue's "canonical only" wording, because pi's ls maps to LS, which Claude Code no longer sends (the MultiEdit skip is now documented as a renaming gap the case-variant diagnostic cannot see). Python: 26 cases covering every branch via subprocess plus in-process assertions on the findings.jsonl records (name/severity/action) for each path.
• Docs — docs/runtimes.md hook-contract bullet and the ADR 0090 note describe the vocabulary as a test-checked reference (not run-time validation), the MCP caveat, and the renaming gap.

Not changed: FULLSEND_TOOL_ALLOWLIST parsing, fail-closed-when-unset, the pi alias maps in pi_agent.go.

Testing

• pytest internal/security/hooks/tool_allowlist_pretool_test.py — 26 pass
• go test ./internal/security/ ./internal/runtime/ — pass; mutation check: adding a bogus name to the Python set fails TestToolAllowlistHook_VocabularyMatchesGo
• go vet, gofmt, ruff check/format, ty, bandit, ADR + docs-link lints (pre-commit on changed files) — clean

Post-script verification

• Branch is not main/master (agent/608-tool-allowlist-diagnostic)
• Secret scan passed (gitleaks — 997021c226272037a8cb69b32c73e14f26073b5e..HEAD)
• PR body secret scan passed (gitleaks — no-git)

[github-actions]

Site preview

Preview: https://8fccba3e-site.fullsend-ai.workers.dev

Commit: 5d908d6efe3baeba09b2dbcc3c784251b1fa85aa

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 12:53 PM UTC · Completed 1:07 PM UTC

Commit: 6b68a6e · View workflow run →

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[fullsend-ai-review]

Looks good to me

Previous run

Review

Findings

Low

• [consumer completeness] internal/security/canonical_tools_test.go:10 — The test for CanonicalClaudeTools only asserts the map is non-empty. There is no test verifying that the hardcoded tool names in security.HookPlan (e.g., "Bash", "WebFetch" in hooks.go) are members of CanonicalClaudeTools. If a future change adds a new HookGroup with a misspelled tool name, nothing catches it until runtime.
  Remediation: Consider adding a test that iterates HookPlan's groups and asserts each non-wildcard tool name is in CanonicalClaudeTools, similar to the pi_agent_test.go pattern.

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 1:18 PM UTC · Ended 1:28 PM UTC

Commit: 3cbd90d · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 1:30 PM UTC · Ended 1:36 PM UTC

Commit: 7688769 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 1:38 PM UTC · Completed 1:55 PM UTC

Commit: 5d908d6 · View workflow run →

[waynesun09]

Verified: internal/security tests pass; the hook this PR modifies (tool_allowlist_pretool.py) is opt-in and disabled fleet-wide today (grepped all fullsend-ai/agents harnesses — zero enable it), so this is currently pure diagnostics/test-coverage with no live runtime effect. Force-enabled it locally (security.sandbox_hooks.tool_allowlist_pretool.enabled + FULLSEND_TOOL_ALLOWLIST) and re-ran the triage agent under the pi runtime end-to-end: hook roster confirmed active, pi's bash/read tool calls passed cleanly through the new normalization check with zero tool_name_unnormalized/allowlist_entry_unnormalized/tool_name_case_collision findings. CI green, review thread resolved.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 2:40 PM UTC · Completed 2:52 PM UTC

Commit: 5d908d6 · View workflow run →

[fullsend-ai-retro]

Retro: PR #6520 — Diagnose Un-normalized Tool Names in Allowlist Hook

Workflow Timeline

Time (UTC)	Stage	Event
12:33	Triage	/fs-triage on issue #608 → agent confirms prerequisites, proposes test cases (run 32639741176)
12:41	Code	/fs-code with detailed constraints → 7-file implementation in ~9 min (run 32640113463)
12:52	PR opened	agent/608-tool-allowlist-diagnostic → PR #6520
12:52–13:07	Review #1	APPROVED with 1 Low finding: HookPlan tool names not tested against CanonicalClaudeTools (run 32640633663)
13:16–13:28	Review #2	Cancelled — superseded by force-push (run 32641863246)
13:28–13:36	Review #3	Cancelled — superseded by force-push (run 32642443289)
13:36–13:55	Review #4	APPROVED — prior finding resolved (run 32642847696)
14:38	Human review	waynesun09 APPROVED with E2E verification (force-enabled hook, ran triage agent under pi runtime, confirmed zero false findings)
14:38	Merged	Squash-merged to main

What Went Well

• Efficient code agent: Comprehensive implementation across Go, Python, and Markdown (7 files, 26+ Python test cases, Go cross-validation tests, docs) in ~9 minutes.
• Actionable review finding: Review agent caught a legitimate consumer-completeness gap (missing HookPlan conformance test) that neither the code agent nor triage agent anticipated.
• Single code iteration: Code agent's output needed only one incremental improvement, not a rework.
• Thorough human verification: Reviewer force-enabled the opt-in hook locally and ran a real triage agent through the pi runtime — integration-level validation the review agent cannot perform.
• Full test coverage: Codecov confirmed all modified lines covered.

Observed Inefficiencies (Covered by Existing Issues)

• 2 cancelled review runs (~20 min CI compute) from rapid force-pushes during manual fixes. Evidence for #4069 and #4960.
• 4 skipped dispatch runs triggered by the bot's own pull_request_review events. Evidence for #5967 and #2994.

Agents Repo

All agents resolved from fullsend-ai/agents@main (commit a0a2c25fac3c).

Proposals filed

• Code agent should add conformance tests for all consumers when introducing a canonical registry (in fullsend-ai/agents)