Skip to content

test: pin what the compiler cannot see, not how it is spelled - #418

Merged
PathGao merged 1 commit into
masterfrom
chore/assert-behaviour-not-spelling
Aug 3, 2026
Merged

test: pin what the compiler cannot see, not how it is spelled#418
PathGao merged 1 commit into
masterfrom
chore/assert-behaviour-not-spelling

Conversation

@PathGao

@PathGao PathGao commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

No production code changes. git status src src-tauri is empty.

The problem, measured

The suite grew quickly, and 674 of its assertions read source text rather than run code. Some of those have no alternative: invoke passes a command name as a plain string that no compiler checks, "this behaviour has one implementation" is not expressible in types, and Svelte components cannot be imported under node (.svelte.ts stores throw $state is not defined under tsx — verified).

But a large share had drifted a layer too low — pinning private identifiers, parameter names, a statement's literal punctuation, and in one case a finally block matched down to three tabs of indentation. A local rename then fails a test, which is friction rather than safety.

The count barely moved: 674 → 668. This is a composition change, not a trim.

Result, measured two ways

Probe Before After
Nine pure private renames (no behaviour change) 5 tests red 0 red
Sixteen injected regressions 15 / 16 caught 16 / 16
Copies of function walk(dir) in scripts/ 3, md5-identical 0
DOMPurify allowlists maintained by hand 2 1
Tests / duration 501 / 3.51s 501 / 3.46s

The one the old suite missed

invoke('render_markdown') moving out of renderMarkdownPreview — bypassing front-matter stripping — while the file still contained exactly one occurrence, ~120 characters after the wrapper's declaration. The old assertion was count === 1 plus a [\s\S]{0,400} proximity window, and it stayed green. Naming the enclosing function of every raw call catches it, with no count and no window.

Where the markers moved to

Was Now Why
/["']dark["']:["']neutral["']/ — the ternary's literal spelling resolveMermaidTheme\( An exported symbol. The old form was evadable by writing if/else
function renderRichContent\(\s*options — the parameter name RenderRichContentOptions The exported type is the contract the export imports
function getLanguage\( — a private 25-line helper return 'plaintext' The Monaco magic string TypeScript cannot check, which any second extension table must also name
const highlightColorMap — a private binding --highlight-color: The CSS custom property — the actual contract with styles.css and FindBar.svelte
indexOf('return processMarkdownHtml(html, filePath, collapsedHeaders);') enclosing-function check Three parameter names and a semicolon were load-bearing
(?:fn|filename) local names + emitted.length === 3 every [[…]] template must contain # The old form let you replace one of the three sites with a broken bare form and stay green at 3
let sanitizedHtml = $derived(sanitizeMarkdownHtml(htmlContent)) read the sink identifier, then assert every bare {@html ident} is that one Adding a second raw injection point now fails; before it did not

Three assertions got stronger, not merely looser. New scripts/sourceTree.ts holds the ex-duplicate walker, the ex-duplicate allowlist, and enclosingFunctionName / callSiteOffsets — the replacements for proximity windows and verbatim-statement matching. It carries the rule in a comment: a marker may be an exported symbol, a cross-language or library magic string, or the literal shape of a fixed defect — never a private identifier.

Kept deliberately

  • The three classes that reproduce defects that really happened: asset.localhost.evil.test prefix forgery, CRLF handling, path case / NFC-NFD. Untouched.
  • Two rules pinning a defect's literal shapegetScrollHeight() - …height (Exclude editor bottom padding from split scroll sync #316) and createElement('iframe') (refactor(preview): drop the dead YouTube iframe copy from the viewer #388), both allowed: []. The criterion protects exactly this.
  • previewRenderRevision.test.ts and viewerDisposal.test.ts pin private locals and are brittle by the criterion — but they are the only coverage of a stale async render overwriting a newer one, and of listeners leaking past component destroy. The logic lives inside .svelte, and no name-free formulation still pins the check. Per "nothing else catches it → do not delete": kept, and flagged here as known-brittle.
  • Exhaustive-count assertions (updateStoredRecentFiles( ×3, discardUnsavedBuffer: true ×2). These close over an enumerated list — "these three, shown above, and no others" — which is the legitimate single-implementation shape, unlike emitted.length === 3, which enumerated nothing.

Verification of the verification

Each of the 18 changes lists which mutation proves the replacement still fires; every mutation was reverted with git checkout. Independently re-run here: a legal rename (npm run check 0 errors, proving it compiles) leaves 501/501 green, and breaking mermaid.render plus planting a duplicate theme ternary in export.ts turns 2 tests red.

npm run check   435 files, 0 errors
npm test        501 / 501
cargo test      131 / 131

Not covered

  • The behavioural half of the sanitizer story still cannot run under node (DOMPurify needs a real DOM); previewSanitize.test.ts still records a browser-measured result in a comment.
  • Two markers remain heuristics a determined rewrite can evade (return 'plaintext', --highlight-color:). Strictly better than the private names they replace, but grep-based single-implementation rules are heuristics, not proofs.
  • lossyDecodeSaveGuard.test.ts has the same smells but is being rewritten in fix(tabs): ask the filesystem whether two paths name the same file #416; left alone to avoid a conflict.
  • Making .svelte logic importable — extracting getLanguage and friends into src/lib/utils/ — would be the real fix for a whole tier of these tests. That is production surgery and out of scope here.

🤖 Generated with Claude Code

The suite grew fast, and 674 of its assertions read source text rather
than run code. Some of those are the only tool available - `invoke`
passes a command name as a plain string, "this behaviour has one
implementation" is not expressible in types, and Svelte components
cannot be imported under node. But a large share had drifted down a
layer, pinning private identifiers, parameter names, a statement's
literal punctuation, and in one case a `finally` block matched down to
three tabs of indentation. Those make a local rename a test failure,
which is friction, not safety.

Nothing was deleted for being redundant without first showing what else
catches the same regression, and the result is measured two ways rather
than asserted:

- nine pure private renames, no behaviour change: 5 tests red before,
  0 after;
- sixteen injected regressions: 15 of 16 caught before, 16 of 16 after.

The one the old suite missed is a front-matter bypass: the raw
`invoke('render_markdown')` moving out of `renderMarkdownPreview` while
the file still contained exactly one occurrence, within the 400-character
proximity window the assertion allowed. Naming the enclosing function
instead of counting occurrences catches it.

Markers moved to the layer that is actually the contract: an exported
symbol, a cross-language magic string the compiler cannot check
(`return 'plaintext'`, `--highlight-color:`), or the literal shape of a
fixed defect. Three byte-identical copies of a directory walker and a
duplicated DOMPurify allowlist collapse into `scripts/sourceTree.ts`,
which carries the rule in a comment.

Kept deliberately: the prefix-forgery, CRLF and path-case tests, which
reproduce defects that really happened; the two rules pinning a defect's
literal shape; and two brittle files that are the only coverage of a
stale async render and a listener leaking past destroy - flagged rather
than removed, since nothing else catches those.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@PathGao
PathGao merged commit 7fea549 into master Aug 3, 2026
4 checks passed
@PathGao
PathGao deleted the chore/assert-behaviour-not-spelling branch August 3, 2026 07:09
PathGao pushed a commit that referenced this pull request Aug 3, 2026
Measured against injected defects, these 14 files detect that a specific line
was edited and nothing else. Each one stayed green while a real defect in its
own stated subject was live; four of them are the only file that greps the line
they pin, so the catch they do provide is the rename detector #418 rules out,
not coverage.

Four assertions inside them were the exception — they pin something no compiler
can see (a Tauri command name, a locale's own dictionary entry, a CSS duration,
an `async fn` whose absence only deadlocks on Windows). Those move into
surviving files rather than being lost:

  findCollapsedMatches   FOLD_TRANSITION_MS vs the styles.css transition
                         -> foldLayout.test.ts
  toolbarCustomization   two per-locale translation tests (they import and run
                         the dictionary) -> i18nCoverage.test.ts
  windowsPdfExport       invoke() name <-> generate_handler! registration
                         -> macosPdfExport.test.ts
  tabContextMenuIsolation  create_transfer_window must stay `async`
                         -> windowOrganization.test.ts

Five files from the same list are kept: each was measured to catch a real
defect that nothing else catches. See the pull request for the per-file table.

562 -> 524 tests; 199 assertions removed, 5 tests added back by the salvage.
Every mutation the suite caught before it still catches, including the three
controls (sanitizer `<style>`, lossy-decode save guard, checkbox toggle line).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PathGao pushed a commit that referenced this pull request Aug 3, 2026
Measured against injected defects, these 14 files detect that a specific line
was edited and nothing else. Each one stayed green while a real defect in its
own stated subject was live; four of them are the only file that greps the line
they pin, so the catch they do provide is the rename detector #418 rules out,
not coverage.

Four assertions inside them were the exception — they pin something no compiler
can see (a Tauri command name, a locale's own dictionary entry, a CSS duration,
an `async fn` whose absence only deadlocks on Windows). Those move into
surviving files rather than being lost:

  findCollapsedMatches   FOLD_TRANSITION_MS vs the styles.css transition
                         -> foldLayout.test.ts
  toolbarCustomization   two per-locale translation tests (they import and run
                         the dictionary) -> i18nCoverage.test.ts
  windowsPdfExport       invoke() name <-> generate_handler! registration
                         -> macosPdfExport.test.ts
  tabContextMenuIsolation  create_transfer_window must stay `async`
                         -> windowOrganization.test.ts

Five files from the same list are kept: each was measured to catch a real
defect that nothing else catches. See the pull request for the per-file table.

562 -> 524 tests; 199 assertions removed, 5 tests added back by the salvage.
Every mutation the suite caught before it still catches, including the three
controls (sanitizer `<style>`, lossy-decode save guard, checkbox toggle line).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PathGao added a commit that referenced this pull request Aug 3, 2026
Measured against injected defects, these 14 files detect that a specific line
was edited and nothing else. Each one stayed green while a real defect in its
own stated subject was live; four of them are the only file that greps the line
they pin, so the catch they do provide is the rename detector #418 rules out,
not coverage.

Four assertions inside them were the exception — they pin something no compiler
can see (a Tauri command name, a locale's own dictionary entry, a CSS duration,
an `async fn` whose absence only deadlocks on Windows). Those move into
surviving files rather than being lost:

  findCollapsedMatches   FOLD_TRANSITION_MS vs the styles.css transition
                         -> foldLayout.test.ts
  toolbarCustomization   two per-locale translation tests (they import and run
                         the dictionary) -> i18nCoverage.test.ts
  windowsPdfExport       invoke() name <-> generate_handler! registration
                         -> macosPdfExport.test.ts
  tabContextMenuIsolation  create_transfer_window must stay `async`
                         -> windowOrganization.test.ts

Five files from the same list are kept: each was measured to catch a real
defect that nothing else catches. See the pull request for the per-file table.

562 -> 524 tests; 199 assertions removed, 5 tests added back by the salvage.
Every mutation the suite caught before it still catches, including the three
controls (sanitizer `<style>`, lossy-decode save guard, checkbox toggle line).

Co-authored-by: PathGao <gaoyanbo@gaoyanbodeMacBook-Air.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant