feat(editor): colour Markdown from the renderer, not from a regex

[PathGao]

Stacked on #684 — that PR is the base branch, and it should merge first. This one only adds the second layer; the palette and its .md scoping come from there.

What this is

The editor's colours now come from the same comrak parse the preview is rendered from, instead of from Monaco's Markdown grammar alone. The rule behind them becomes exact: if it is coloured, it renders.

What that fixes, measured by typing the line and reading the rendered tokens back:

source	before	after
- [x] task	[x] purple — the grammar reads the brackets as a link	[x] blue, as markup
~~gone~~ ==lit== ++new++	no colour at all — the grammar has never heard of them	struck / highlighted / underlined
[[a page]], $E=mc^2$	no colour	wikilink and math, delimiters apart from content
snake_case_word	italic — the grammar's \b_[^_]+_\b matches inside the word	plain, because CommonMark renders no emphasis there
**unclosed	not coloured (correct by luck)	not coloured (correct by construction)
## Title	one keyword for the whole line	## as markup, Title as bold content

Mechanism

Monaco merges the two layers per attribute, not per range (sparseTokensStore.js:136): a semantic token declares which of foreground/bold/italic/underline/strikethrough it owns, the grammar keeps the rest, and any range the parse says nothing about is left entirely alone. That is what makes this additive rather than a second source of truth — the grammar still paints instantly on every keystroke, and these spans refine it a tick later.

Three things the parse forced, each a defect if missed:

The raw buffer is parsed, not the preprocessed one. convert_markdown rewrites wikilinks, internal embeds and parenthesized autolinks before comrak sees the text, and those rewrites change a line's length ([[a]] → [a](a.md)). Line numbers survive — that is the contract data-sourcepos and the task-checkbox toggle rest on — but columns do not, and columns are the whole point. The cost is that comrak cannot see the app's own syntaxes in the raw text, so wikilinks are scanned separately and math reuses find_math_spans, the same scanner mask_math_spans uses to decide what KaTeX will typeset.

Columns are converted to UTF-16 in Rust. comrak reports 1-based byte offsets; Monaco counts UTF-16 code units. For 中文 **粗体** the strong marker is at byte 7 and unit 3, and a span computed in bytes lands mid-character, where Monaco drops it silently. Emoji make it three counting systems.

Semantic rules spell out their font styles. standaloneThemeService.getTokenStyleMetadata returns bold and italic as booleans rather than "not set", so a semantic token always claims those bits — a rule that stays silent about bold switches it off for its range. That is the opposite of the grammar rules, where silence means "leave it alone", and it is why SEMANTIC_TOKEN_ROLES names a fontStyle wherever one matters.

Also: 'semanticHighlighting.enabled': true is set explicitly. StandaloneTheme.semanticHighlighting is hard-coded false, so 'configuredByTheme' cannot reach it and the provider would never be asked.

Scope

The grammar keeps its rules. Deleting them would leave text unpainted for the tick between a keystroke and the parse; keeping them costs nothing, because the semantic layer overrides where it has an opinion.

One parse is cached per model version, because Monaco asks on scroll as well as on edit. Measured on this machine, release build, median of 20: 0.67 ms for a 527-line document, 2.06 ms for 3337 lines, 24 ms for 1.7 MB — of which the AST walk is 1–4%; the rest is the parse the preview already pays for. If the 1.7 MB case ever matters, registerDocumentRangeSemanticTokensProvider asks only for the visible range and is a drop-in.

Not covered by the parse, so not coloured: nothing in a fenced code block (the embedded language grammar keeps it), and reference-style link definitions.

Tests

src-tauri/src/semantic.rs — 8 unit tests over the extractor: the heading split, UTF-16 columns for CJK and for an astral character, the task checkbox not being a link, the four syntaxes the grammar cannot see, snake_case_word and **unclosed producing nothing, fences marked without touching the code between them, and — the one that guards the encoding — spans never overlapping, since Monaco's delta encoding turns an overlap into a token on the wrong line.

scripts/semanticTokens.test.ts — 6 tests over the three seams the compiler cannot see: every kind semantic.rs emits is in the legend, every legend type has a theme rule and vice versa, the font styles are spelled out, the delta encoding, an unknown kind being dropped rather than mapped to whatever construct sits at that index, and the enable switch being present.

Verification

npm audit             0 vulnerabilities
npm run check         808 files, 0 errors, 0 warnings
npm test              944 pass, 0 fail
npm run test:vitest   42 files, 376 pass
cargo test            156 pass

The before/after table was measured in Chromium against the dev server with window.__TAURI_INTERNALS__ stubbed, feeding the provider the exact spans the real extractor prints for that text, then reading getComputedStyle off the rendered tokens: ## and its title blue at weight 700, [x] blue rather than purple, ~~删除~~ muted and struck, ==高亮== amber, snake_case_word plain.

What that does not cover: the IPC hop itself. In the browser the command is a stub — the extractor is verified by its own tests and by running it over fixtures, but nobody has yet watched Monaco ask Rust and repaint in a real window. That is what the desktop build is for.