Skip to content

feat(editor): a formula reads as markup around operands - #687

Merged
PathGao merged 1 commit into
masterfrom
feat/math-token-colours
Aug 19, 2026
Merged

feat(editor): a formula reads as markup around operands#687
PathGao merged 1 commit into
masterfrom
feat/math-token-colours

Conversation

@PathGao

@PathGao PathGao commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Two changes to how a formula looks in the editor, one in the theme and one in the extractor. Stacked on nothing — #686 is merged and this sits on master.

What was wrong

math.marker and math both resolved to the code role, so the split the semantic layer exists to make was invisible: the $ and the formula were the same green. And the body was one flat span, so \frac and the a it applies to arrived identical.

source before after
$…$, $$…$$ same green as the body markup blue, like ## and >
formula body green, upright green, italic
\frac, \alpha, \,, \\ body green markup blue
{, }, _, ^, +, = body green body green, deliberately

Why those choices

A control sequence is markup. \frac is to a and b what ## is to a title, so it takes the marker modifier and lands on the rule that already exists. No new legend entry, no new palette colour.

Braces and scripts stay with the operands. They are markup too, and marking them was the first thing I tried: \frac{\sqrt{\pi}}{2} comes out almost entirely blue, with \pi and 2 the only green left, and the command names stop standing out. VS Code's md-math grammar ends up in the same place from the other direction — it scopes braces and operators, and the default themes give those scopes no colour at all.

The body is italic. The one thing about a formula a colour cannot say. TeX typesets variables italic, so the source and the rendered output agree at a glance. Obsidian pairs the italic with a monospace family; ITokenThemeRule has no fontFamily and Monaco's mtkN class is a per-style index rather than a per-construct one, so only half of that is reachable from a theme.

The body is covered with no gaps. A requirement, not tidiness: a range no semantic token claims keeps the Monarch grammar's colour, and the grammar has never heard of $. It reads the two underscores in x_i + y_j as an emphasis run and italicises the middle of the formula — the defect Zettlr#276 describes. the_formula_body_is_covered_whole is the test that holds it.

comrak's own math extension is off in markdown_options, so math_spans is the only thing that claims these ranges and nothing competes with the finer spans.

Cost

Two dense inline formulas on every one of 500 lines — a document far past anything real:

spans median parse
before 3000 3.40 ms
after 12000 4.44 ms

Prose is unchanged at 1.3 ms, and one parse is still cached per model version.

Tests

semantic.rs — a control sequence carrying the marker modifier, including the single-character \,; braces and scripts staying with the operands; the body covered with no gaps.

editorTheme.test.ts — the delimiters resolving to the same colour as every other marker and not to the body's, the body keeping the code colour, and the italic being spelled out.

Verification

npm run check         809 files, 0 errors, 0 warnings
npm test              948 pass
npm run test:vitest   376 pass
cargo test            160 pass
cargo fmt --check     clean
cargo clippy          -D warnings, clean

🤖 Generated with Claude Code

Both math rules resolved to the same colour, so the split between the
`$` delimiters and the body — the thing the semantic layer exists to
make — was invisible on screen. And the body was one flat run: `\frac`
and the `a` it is applied to arrived identical.

The delimiters take the markup colour every other marker has. Inside
them, a control sequence is markup too — `\frac` is to `a` and `b` what
`##` is to a title — so `math_spans` gives it the same `marker`
modifier and it lands on the same rule. Braces, `_` and `^` deliberately
stay with the operands: they are markup as well, but marking them leaves
almost nothing in the operand colour and the command names stop standing
out. VS Code's `md-math` grammar ends up in the same place, by way of a
default theme that gives its bracket and operator scopes no colour.

The body is also italic now, which is the one thing about a formula a
colour cannot say: TeX typesets variables italic, so the source and the
rendered output agree at a glance. Obsidian pairs that italic with a
monospace family; `ITokenThemeRule` has no `fontFamily`, so only half of
that is reachable from a Monaco theme.

The body is covered with no gaps, which is a requirement rather than
tidiness. A range no semantic token claims keeps the Monarch grammar's
colour, and the grammar has never heard of `$` — it reads the two
underscores in `x_i + y_j` as an emphasis run and italicises the middle
of the formula. That is the defect Zettlr#276 describes.

Measured on a document with two dense inline formulas on every one of
500 lines: spans go from 3000 to 12000 and the parse from 3.40 ms to
4.44 ms, still cached per model version. Prose is unchanged at 1.3 ms.
@PathGao
PathGao merged commit 98133e9 into master Aug 19, 2026
4 checks passed
@PathGao
PathGao deleted the feat/math-token-colours branch August 19, 2026 17:36
PathGao pushed a commit that referenced this pull request Aug 19, 2026
The alias reached the content and stopped there, so a theme coloured a
word and the app went on colouring the markup around it: `~~` in one
grey and the struck word in another, `##` blue against a red title, `[`
purple against a cyan link text.

Insertion order is why. Rules are sorted by name, so `strike` lands
before `strike.marker`, and a child node is cloned from its parent at
the moment it is created. The theme's `strike` updated the parent, then
the app's `strike.marker` — longer, therefore last — created the child
from that and overwrote the foreground with its own. The base was
supposed to sit underneath the theme and was sitting on top of it for
every marker.

An alias now names `<kind>.marker` beside `<kind>`, which puts the
theme's rule after the app's on both.

Swept the rest of the pairs the same way, by resolving every type and
its marker through Monaco's trie: with this in, the only construct whose
marker and content differ under an imported theme is maths, which is
#687 deliberately colouring `\frac` apart from its operands, and the
bare `list`, `task`, `quote` and `table` types no extractor span uses.
The built-in themes come out uniform on every pair but that same one.
PathGao added a commit that referenced this pull request Aug 19, 2026
…ayer (#689)

* fix(themes): an imported theme keeps its colours under the semantic layer

A semantic token whose type the active theme does not name does not fall
back to the grammar's colour — it resolves to the theme's *root* rule,
whose foreground is a real colour id rather than "none", and
`sparseTokensStore` then masks the grammar's foreground out and paints
that plain default over it. `getTokenStyleMetadata` also reports the four
font styles as booleans rather than "not set", so bold, italic,
underline and strikethrough are taken over and switched off with it.

The two built-in themes are handed `semanticTokenRules()` and name every
type. An imported VS Code theme was handed `monacoTokenRules()` alone,
which renames scopes to the names the *grammar* emits. Of the kinds the
extractor sends, exactly one matched — `strong`, and only because #676's
`markup.bold` alias happens to collide with the legend's spelling. So
since the semantic layer shipped, an imported theme has been losing its
heading, list, quote, table, inline-code, link and image colours to the
default foreground, and `~~strike~~`, `==highlight==` and `++insert++`
their font styles with them.

Two halves. The alias table now carries the construct names beside the
grammar ones, so a theme's own `markup.italic`, `markup.strikethrough`,
`markup.heading`, `markup.quote`, `markup.list` and the rest reach the
layer that reports them — which is the part of #676 that could not be
answered while the grammar decided the colours, since its tokenizer has
no name for a strikethrough at all. And `importedThemeRules` puts the
app's semantic rules underneath the theme's, so a construct with no
`markup.*` scope of its own — a task checkbox, a table, a wikilink,
maths — takes a deliberate colour instead of the default.

The test resolves every kind the extractor emits through Monaco's own
theme trie and fails on any that lands on the fallback rule, because
this failure is invisible from either file alone: the rules look
complete on one side and the kinds look covered on the other.

* fix(themes): an imported theme colours a construct's markers too

The alias reached the content and stopped there, so a theme coloured a
word and the app went on colouring the markup around it: `~~` in one
grey and the struck word in another, `##` blue against a red title, `[`
purple against a cyan link text.

Insertion order is why. Rules are sorted by name, so `strike` lands
before `strike.marker`, and a child node is cloned from its parent at
the moment it is created. The theme's `strike` updated the parent, then
the app's `strike.marker` — longer, therefore last — created the child
from that and overwrote the foreground with its own. The base was
supposed to sit underneath the theme and was sitting on top of it for
every marker.

An alias now names `<kind>.marker` beside `<kind>`, which puts the
theme's rule after the app's on both.

Swept the rest of the pairs the same way, by resolving every type and
its marker through Monaco's trie: with this in, the only construct whose
marker and content differ under an imported theme is maths, which is
#687 deliberately colouring `\frac` apart from its operands, and the
bare `list`, `task`, `quote` and `table` types no extractor span uses.
The built-in themes come out uniform on every pair but that same one.

* test(themes): drive the marker check off the alias table

The eight scope/kind pairs were written out in the test, which covers
what broke and nothing that might. The defect appears whenever the app's
base names a *longer* token than an alias does — so a newly added alias
is exactly the case a hardcoded list cannot see, and adding one is the
likeliest way to reintroduce this.

Iterating the table means an entry is covered the moment it exists. The
generated colour is per kind rather than per scope, because two scopes
can name the same construct — `markup.inline.raw` and `markup.raw.inline`
are both inline code — and giving those two different colours would only
have tested which of them sorts last.

* refactor(themes): resolve the theme into the base instead of layering under it

The previous commit fixed the markers by having an alias name
`<kind>.marker` beside `<kind>`. That works only because the alias table
knows how the base is spelled, which is the coupling that produced the
bug in the first place — the base and the table each looked complete on
their own.

Monaco has no notion of one rule set overriding another. Rules are
sorted by name and merged into a trie where a child is cloned from its
parent when it is created and never revisited. Between two rules of the
same name, array order decides and the theme wins; between `strike` and
`strike.marker`, the trie shape decides, the longer name is inserted
last and overwrites the child, and which set a rule came from does not
enter into it. Layering by array order holds for exactly as long as the
two sets happen to name the same tokens, and inverts silently the moment
the base names a longer one.

So the base is rewritten rather than layered under: every rule the app
declares for a construct the theme named takes the theme's colour before
Monaco sees any of them. The base's shape stops mattering — marker,
content, or a name not invented yet. The alias table goes back to saying
only which construct a scope means.

Font styles stay the base's: `strike` has to be struck through and
`insert` underlined whatever colour they take, and a theme that spells
out a style still applies it through its own rule below.

The scope is matched by the same prefix rule as the grammar aliases, so
`markup.bold.markdown` — the commoner spelling — resolves like the bare
form. The test now qualifies every other scope to cover both.

---------

Co-authored-by: PathGao <gaoyanbo@gaoyanbodeMacBook-Air.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant