test(fuzz): expand property tests — tool schemas, execute(), MCP prefix, markdown - #141
Merged
Merged
Conversation
… prefix, markdown Per Dylan's "expand fuzz testing" follow-up to the bug-finding-tier discussion. Mutation testing on the full suite is too slow for GHA; property tests are the cheap-and-fast equivalent. This adds adversarial pressure to the messaging-tool surface, the MCP prefix normalizer, and the markdown→HTML converter — all paths Talon hits on every outbound message. +45 new fuzz tests, 12 → 57 total. Runtime: ~5s at 1000 iterations, will be ~50s at the 10000-iteration CI default. Well under budget. ## New coverage ### `stripMcpPrefix()` + `isTurnTerminator()` The boundary normalizer between bare tool names (`end_turn`) and MCP- prefixed (`mcp__telegram-tools__end_turn`). PR #122 lived 8 weeks because three call sites compared bare-only and the prefixed shape never matched. Tests: - never throws on arbitrary strings - input without `mcp__` prefix returns unchanged - synthesized `mcp__<server>__<tool>` always reduces to `<tool>` - `isTurnTerminator` agrees on bare/prefixed equivalence (anchors PR #122 regression — if a future change re-introduces the strict-equality pattern, this fails fast) ### Messaging-tool zod schemas (~12 × 2 = 24 tests) For every tool in the registry, two properties: - `safeParse` never throws on arbitrary objects - primitives / null / undefined always return `success: false` Catches schema regressions: a future change that turns a `z.string()` into a discriminated union missing a case would fail here. ### Messaging-tool `execute()` (~12 tests) For every tool, generate dictionary input via `fc.dictionary` and call `tool.execute(input, fakeBridge)`. Asserts: - never throws unhandled (catches and tolerates Error throws — schema validation is allowed to fail loudly) - returns object-shaped result or undefined - bridge fan-out is bounded (≤2 calls per execute) ### `markdownToTelegramHtml` + `escapeHtml` Every outbound message flows through these. Tests: - never throws on arbitrary strings - never throws on long arbitrary strings (size-stressed, unicode) - never throws on heavily nested markdown (depth 1-20 nesting of `*`, `_`, `` ` ``) - `escapeHtml` always escapes `<`, `>`, `&`, `"`, `'` (no raw chars in output after entity-stripping) ## Local verification - `npx vitest run src/__tests__/fuzz.test.ts` — 57 / 57 pass at default 100 iterations, 2.3s - `FAST_CHECK_NUM_RUNS=1000` — 57 / 57 pass, 5.3s - tsc clean, prettier clean, oxlint clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Per Dylan's "expand fuzz testing" call from chat. Mutation testing on the full suite is too slow for GHA; property tests are the cheap-and-fast equivalent. This adds adversarial pressure to the messaging-tool surface, the MCP prefix normalizer, and the markdown→HTML converter — all paths Talon hits on every outbound message.
+45 new fuzz tests, 12 → 57 total. Runtime: ~5s at 1000 iterations, will be ~50s at the 10000-iteration CI default. Well under budget.
New coverage
stripMcpPrefix()+isTurnTerminator()The boundary normalizer between bare tool names (
end_turn) and MCP-prefixed (mcp__telegram-tools__end_turn). PR #122 lived 8 weeks because three call sites compared bare-only and the prefixed shape never matched. Tests:mcp__prefix returns unchangedmcp__<server>__<tool>always reduces to<tool>isTurnTerminatoragrees on bare/prefixed equivalence — regression anchor for PR fix(claude-sdk): terminate SDK loop on end_turn (MCP-prefix match + PostToolBatch hook) #122Messaging-tool zod schemas (~12 × 2 = 24 tests)
For every tool in the registry, two properties:
safeParsenever throws on arbitrary objectssuccess: falseMessaging-tool
execute()(~12 tests)For every tool, generate dictionary input and call
tool.execute(input, fakeBridge). Asserts:markdownToTelegramHtml+escapeHtmlEvery outbound message flows through these:
*_`)escapeHtmlalways escapes<,>,&,\",'Test plan
tsc --noEmitcleanprettier --checkcleanoxlintcleanWhat's next on this thread
Property/fuzz coverage gaps still worth filling:
🤖 Generated with Claude Code