Split tokenizing into ktsu.SyntaxHighlighting and highlight embedded languages - #357
Merged
matt-edmondson merged 4 commits intoSep 8, 2026
Merged
Conversation
…mbedded languages Syntax highlighting was one package that both classified code and drew it with Dear ImGui. Tokenizing has nothing to do with a graphics API, so it now lives in ktsu.SyntaxHighlighting: languages, tokenizers, themes and the token kinds, with no ImGui reference anywhere, ready to move to its own repository. ktsu.ImGui.SyntaxHighlighting keeps the drawing half and forwards to it, so ImGuiSyntaxHighlighting.Render and .Highlight are unchanged; HighlightedCode.Render moves to an extension method, since the tokenized type itself no longer knows about ImGui. On top of that split, comments and strings are now searched for the language written inside them. XML doc comments, JSON payloads and SQL queries are all written inside a host language's literals, and a lexer that stops at the quote leaves them one flat color. Each LanguageDefinition carries EmbeddedLanguages rules; the built-ins get JSON, markup and SQL, plus XML doc comments for C#. Four things keep this safe to leave on by default: - Recognition is strict. LooksLikeJson parses rather than pattern matches, markup must open with a tag and end with '>', and SQL needs an opening keyword *and* a second one, in a string only, so prose is not recolored. - Embedded text the inner tokenizer does not classify keeps its host's kind, so prose between doc comment tags still reads as a comment. - Escapes are resolved for the tokenizer but every token is re-sliced from the original text through an index map, and the expander verifies the run reconstructs the host token before keeping it, so the rendered text is never rewritten. - Embedding is one level deep, so definitions cannot cycle. A '// lang=json' hint comment names the language of the next string literal outright, for snippets no recognizer can catch. Also fixes the highlight cache keying by language name, which served stale lines after a definition was re-registered under the same name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019uHKuTn4tHvuCwCFZ9UYVW
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Two of the four CodeQL comments on #357 are actionable: - TryResolve's foreach filtered its sequence implicitly; the AppliesTo test moves into a Where, and the comment now says why a matching rule naming an unregistered language falls through to the next rule rather than failing. - UnescapedBody.Of leaned on a null-forgiving rule! after proving non-null through a separate bool. It now returns early on null (NeedsUnescaping already answers false when neither escape mechanism applies) and caches the escape, close and doubled-close members in locals, so no nullable member is dereferenced in the loop. The other two flag foreach loops over string[] whose bodies compare against a ReadOnlySpan<char> parameter. A Where predicate cannot capture one — CS9108, "cannot use parameter that has ref-like type inside a lambda" — so the suggestion does not compile there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019uHKuTn4tHvuCwCFZ9UYVW
Moving TryResolve's AppliesTo test into a Where left the loop body a single if, so the same "missed opportunity to use Where" finding fired again on the rewritten loop. Reshaping it a third time would keep the shape that draws the finding, so the loop is gone: the rule that claims the body and the language it names are now selected together, and the first that resolves is the match. Behavior is unchanged, including the part worth stating — a rule that claims the body but names a language the registry does not know falls through to the next rule rather than ending the search. That was implicit in the loop and had no test; it has one now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019uHKuTn4tHvuCwCFZ9UYVW
The hint is read from a comment's body, so /* lang=json */ and an inline /* language=sql */ beside the string both work, but only the // form had a test and only the // form was in the README. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019uHKuTn4tHvuCwCFZ9UYVW
|
matt-edmondson
deleted the
claude/embedded-language-highlighting-so2k74
branch
September 8, 2026 05:01
matt-edmondson
pushed a commit
that referenced
this pull request
Sep 8, 2026
…-354-7ftjju #356 and #357 landed: ImGuiNodeEditor was renamed to ImGui.NodeEditor, and tokenizing was split out of ImGui.SyntaxHighlighting into its own SyntaxHighlighting library. Neither touches the image decoder. The only conflict was CLAUDE.md's Libraries list, where main rewrote both the ImGui.App and ImGui.Widgets entries. Took main's text for both and re-applied just this branch's addition: the sentence on ImGui.App pointing at the self-contained image decoding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DCFaH3HSDMKHTmrQB7LcSP
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes, in this order:
1. The renderer-agnostic half is its own library.
ktsu.SyntaxHighlightingnow holds the languages, tokenizers, token kinds and themes — no ImGui, no graphics API, no third-party parser — so it can move to its own repository unchanged.ktsu.ImGui.SyntaxHighlightingkeeps the drawing half (CodeRenderer,SyntaxHighlightConfig,SyntaxColors,ScopedCodeFont) and references it.Public API is unchanged apart from namespaces and one member:
ImGuiSyntaxHighlighting.Render/.Highlightstill work as before (Highlightforwards toSyntaxHighlighter.Highlight), andHighlightedCode.Render(config)still compiles — it is now an extension method in the ImGui package, because the tokenized type no longer knows what ImGui is. Callers ofLanguageDefinition,TokenKind,SyntaxThemeetc. addusing ktsu.SyntaxHighlighting;.2. Comments and strings are searched for the language written inside them. XML doc comments, JSON payloads and SQL queries are all written inside a host language's literals, and a lexer that stops at the quote leaves them one flat color:
Each
LanguageDefinitioncarriesEmbeddedLanguages, a list ofEmbeddedLanguageRuletried in order. The built-in programming languages get JSON, markup and SQL; JSON, YAML and SQL get JSON and markup; C# also gets "doc comments are XML". An empty list turns it off, soBuiltInLanguages.CSharp with { EmbeddedLanguages = [] }is the opt-out.Why it is safe to leave on by default
LooksLikeJsonparses — a fragment, trailing text, or a brace-heavy sentence is rejected. Markup must open with a tag and end with>. SQL must open with a statement keyword and use a second one, and its rule applies to string literals only, so "Update the cache and carry on" in a comment is never a query.Plaintoken from the inner tokenizer is re-emitted asComment/DocComment/StringLiteral, so prose between doc comment tags still reads as a comment and a false positive costs a few punctuation glyphs, not a paragraph.{"a": 1}where the source holds{\"a\": 1}, but every token is re-sliced from the original through an index map, and the expander verifies the run reconstructs the host token before keeping it, falling back to the unexpanded token otherwise.Also
Fixes the highlight cache keying by language name: re-registering a definition under an existing name served lines tokenized by the old rules. It now keys on the definition instance.
Testing
tests/SyntaxHighlighting.Tests/(moved fromtests/ImGui.SyntaxHighlighting.Tests/, now referencing the core package only): 70 tests pass, including a newEmbeddedLanguageTestscovering each recognizer, the escaped-JSON index mapping, hint comments, the one-level-deep bound, and the negative cases.tests/ImGuiSyntaxHighlightingDemo.UITests/: 12 pass, including the demo's new Embedded tab.ImGuiAppDemo.UITests,ImGuiWidgetsDemo.UITestsandImGuiMarkdownDemo.UITests, allUnknownImageFormatExceptionfrom Git LFS pointer files that were not resolved in this sandbox — untouched by this change and expected to pass in CI.Docs
Root
README.md,CLAUDE.md(new Embedded languages architecture section),DESCRIPTION.md,TAGS.md,ImGui.SyntaxHighlighting/README.md, and a newSyntaxHighlighting/README.mdfor the package that is destined for its own repo.🤖 Generated with Claude Code
https://claude.ai/code/session_019uHKuTn4tHvuCwCFZ9UYVW
Generated by Claude Code