Skip to content

test(fuzz): expand property tests — tool schemas, execute(), MCP prefix, markdown - #141

Merged
claudiusthebot merged 1 commit into
mainfrom
feat/expand-fuzz-tests
May 10, 2026
Merged

test(fuzz): expand property tests — tool schemas, execute(), MCP prefix, markdown#141
claudiusthebot merged 1 commit into
mainfrom
feat/expand-fuzz-tests

Conversation

@claudiusthebot

Copy link
Copy Markdown
Collaborator

Summary

Per Dylan's "expand fuzz testing" call from chat. Mutation testing on the full suite is too slow for GHA; property tests are the cheap-and-fast equivalent. This adds adversarial pressure to the messaging-tool surface, the MCP prefix normalizer, and the markdown→HTML converter — all paths Talon hits on every outbound message.

+45 new fuzz tests, 12 → 57 total. Runtime: ~5s at 1000 iterations, will be ~50s at the 10000-iteration CI default. Well under budget.

New coverage

stripMcpPrefix() + isTurnTerminator()

The boundary normalizer between bare tool names (end_turn) and MCP-prefixed (mcp__telegram-tools__end_turn). PR #122 lived 8 weeks because three call sites compared bare-only and the prefixed shape never matched. Tests:

Messaging-tool zod schemas (~12 × 2 = 24 tests)

For every tool in the registry, two properties:

  • safeParse never throws on arbitrary objects
  • primitives / null / undefined always return success: false

Messaging-tool execute() (~12 tests)

For every tool, generate dictionary input and call tool.execute(input, fakeBridge). Asserts:

  • never throws unhandled (Error throws tolerated — schema validation is allowed to fail loudly)
  • returns object-shaped result or undefined
  • bridge fan-out is bounded (≤2 calls per execute)

markdownToTelegramHtml + escapeHtml

Every outbound message flows through these:

  • never throws on arbitrary strings
  • never throws on long arbitrary strings (size-stressed)
  • never throws on heavily nested markdown (depth 1-20 of * _ `)
  • escapeHtml always escapes <, >, &, \", '

Test plan

  • tsc --noEmit clean
  • prettier --check clean
  • oxlint clean
  • 57/57 fuzz tests pass at default 100 iterations (2.3s)
  • 57/57 fuzz tests pass at 1000 iterations (5.3s)
  • CI run at 10000 iterations (the existing fuzz job's setting)

What's next on this thread

Property/fuzz coverage gaps still worth filling:

  • Session state mutations (write/read invariants)
  • Config schema fuzz (TalonConfig zod)
  • Stream-event processing (assistant-message content blocks)
  • Chat-id parsing (numeric / string / Teams-style "19:abc")

🤖 Generated with Claude Code

… prefix, markdown

Per Dylan's "expand fuzz testing" follow-up to the bug-finding-tier
discussion. Mutation testing on the full suite is too slow for GHA;
property tests are the cheap-and-fast equivalent. This adds adversarial
pressure to the messaging-tool surface, the MCP prefix normalizer, and
the markdown→HTML converter — all paths Talon hits on every outbound
message.

+45 new fuzz tests, 12 → 57 total. Runtime: ~5s at 1000 iterations,
will be ~50s at the 10000-iteration CI default. Well under budget.

## New coverage

### `stripMcpPrefix()` + `isTurnTerminator()`
The boundary normalizer between bare tool names (`end_turn`) and MCP-
prefixed (`mcp__telegram-tools__end_turn`). PR #122 lived 8 weeks
because three call sites compared bare-only and the prefixed shape
never matched. Tests:
- never throws on arbitrary strings
- input without `mcp__` prefix returns unchanged
- synthesized `mcp__<server>__<tool>` always reduces to `<tool>`
- `isTurnTerminator` agrees on bare/prefixed equivalence (anchors PR
  #122 regression — if a future change re-introduces the strict-equality
  pattern, this fails fast)

### Messaging-tool zod schemas (~12 × 2 = 24 tests)
For every tool in the registry, two properties:
- `safeParse` never throws on arbitrary objects
- primitives / null / undefined always return `success: false`

Catches schema regressions: a future change that turns a `z.string()`
into a discriminated union missing a case would fail here.

### Messaging-tool `execute()` (~12 tests)
For every tool, generate dictionary input via `fc.dictionary` and call
`tool.execute(input, fakeBridge)`. Asserts:
- never throws unhandled (catches and tolerates Error throws — schema
  validation is allowed to fail loudly)
- returns object-shaped result or undefined
- bridge fan-out is bounded (≤2 calls per execute)

### `markdownToTelegramHtml` + `escapeHtml`
Every outbound message flows through these. Tests:
- never throws on arbitrary strings
- never throws on long arbitrary strings (size-stressed, unicode)
- never throws on heavily nested markdown (depth 1-20 nesting of
  `*`, `_`, `` ` ``)
- `escapeHtml` always escapes `<`, `>`, `&`, `"`, `'` (no raw chars
  in output after entity-stripping)

## Local verification

- `npx vitest run src/__tests__/fuzz.test.ts` — 57 / 57 pass at default
  100 iterations, 2.3s
- `FAST_CHECK_NUM_RUNS=1000` — 57 / 57 pass, 5.3s
- tsc clean, prettier clean, oxlint clean
@claudiusthebot
claudiusthebot enabled auto-merge (squash) May 10, 2026 10:57
@claudiusthebot
claudiusthebot merged commit ffe973a into main May 10, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant