Skip to content

Some fixes to tokenize #3942

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DmitrySharabin merged 3 commits into from
May 25, 2025
Merged

Some fixes to tokenize #3942

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dc230c7
Not every pattern is an instance of `RegExp`
DmitrySharabin May 25, 2025
1a383af
Work with resolved grammars
DmitrySharabin May 25, 2025
41093b8
Delete `$rest` after the grammar is resolved
DmitrySharabin May 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/core/tokenize/match.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ export function _matchGrammar (
// Without the global flag, lastIndex won't work
flagsToAdd += 'g';
}
if (pattern.source.includes('(?<') && pattern.hasIndices === false) {
if (pattern.source?.includes('(?<') && pattern.hasIndices === false) {
// Has named groups, we need to be able to capture their indices
flagsToAdd += 'd';
}
Expand Down
3 changes: 2 additions & 1 deletion src/core/tokenize/util.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ export function resolve (
let ret = reference ?? undefined;

if (typeof ret === 'string') {
ret = prism.languageRegistry.getLanguage(ret)?.grammar;
ret = prism.languageRegistry.getLanguage(ret)?.resolvedGrammar;
Copy link
Member Author

@DmitrySharabin DmitrySharabin May 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still not fully resolving cases like:

{
  id: "foo",
  require: bar
  grammar: {
    token: {
      ...
      inside: "bar"
    }
  }
}

The bar language is imported and can be accessed via the languages array associated with the foo language. However, languages["bar"] is evaluated lazily. Until it's accessed for the first time, the bar language won't be added to the languageRegistry, and resolve(prism, "bar") will always return undefined.

Honestly, I don't know how to solve it without providing some extra info on the grammar we resolve. 🤷‍♂️

Copy link
Member

@LeaVerou LeaVerou May 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll need to traverse the grammar for resolvedGrammar and resolve anything that can be resolved without context. That will help with many things, e.g. self as well. It won't help with optional languages though. But we can replace it with a getter that overwrites itself.

Copy link
Member Author

@DmitrySharabin DmitrySharabin May 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a plan 👍

}

if (typeof ret === 'function' && ret.length === 0) {
Expand All @@ -20,6 +20,7 @@ export function resolve (

if (typeof ret === 'object' && ret.$rest) {
let restGrammar = resolve.call(prism, ret.$rest) ?? {};
delete ret.$rest;

if (typeof restGrammar === 'object') {
ret = { ...ret, ...restGrammar };
Expand Down