feat(engine-js): update deps, support more languages (#890)

Co-authored-by: Anthony Fu <[email protected]>
shikijs · Jan 15, 2025 · d82af15 · d82af15
1 parent 07da2f3
commit d82af15
Show file tree

Hide file tree

Showing 7 changed files with 30 additions and 29 deletions.
diff --git a/docs/guide/regex-engines.md b/docs/guide/regex-engines.md
@@ -56,8 +56,6 @@ const html = shiki.codeToHtml('const a = 1', { lang: 'javascript', theme: 'nord'
 
 The advantages of using the JavaScript engine are that it doesn't require loading a large WebAssembly file for Oniguruma and it is faster for some grammars (since the regular expressions run as native JavaScript).
 
-Although the JavaScript engine's Oniguruma emulation is quite robust, it's not guaranteed that the highlighting will be 100% the same in some edge cases. Additionally, a few grammars are unsupported.
-
 Please check the [compatibility table](/references/engine-js-compat) for the support status of languages you are using.
 
 The JavaScript engine is strict by default, and will throw an error if it encounters a pattern that it cannot convert. If mismatches are acceptable and you want best-effort results for unsupported grammars, you can enable the `forgiving` option to suppress any conversion errors:
@@ -68,7 +66,7 @@ const jsEngine = createJavaScriptRegexEngine({ forgiving: true })
 ```
 
 ::: info
-If you run Shiki on Node.js (or at build time) and bundle size or WebAssembly support is not a concern, we still recommend using the Oniguruma engine for best results.
+If you run Shiki on Node.js (or at build time) and bundle size or WebAssembly support is not a concern, we still recommend using the Oniguruma engine.
 
 The JavaScript engine is best when running in the browser and in cases when you want to control the bundle size.
 :::

diff --git a/docs/references/engine-js-compat.md b/docs/references/engine-js-compat.md
@@ -2,19 +2,19 @@
 
 Compatibility reference of all built-in grammars with the [JavaScript RegExp engine](/guide/regex-engines#javascript-regexp-engine).
 
-> Generated on Monday, January 13, 2025
+> Generated on Tuesday, January 14, 2025
 >
-> Version `1.26.1`
+> Version `1.27.0`
 >
-> Runtime: Node.js v22.12.0
+> Runtime: Node.js v22.11.0
 
 ## Report Summary
 
 |                 |                       Count |
 | :-------------- | --------------------------: |
 | Total Languages |                         219 |
-| Supported       | [212](#supported-languages) |
-| Mismatched      |  [1](#mismatched-languages) |
+| Supported       | [213](#supported-languages) |
+| Mismatched      |  [0](#mismatched-languages) |
 | Unsupported     | [6](#unsupported-languages) |
 
 ## Supported Languages
@@ -147,6 +147,7 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
 | move               | ✅ OK           |               120 |               - |      |
 | narrat             | ✅ OK           |                34 |               - |      |
 | nextflow           | ✅ OK           |                32 |               - |      |
+| nginx              | ✅ OK           |               378 |               - |      |
 | nim                | ✅ OK           |              1130 |               - |      |
 | nix                | ✅ OK           |                80 |               - |      |
 | nushell            | ✅ OK           |                81 |               - |      |
@@ -204,7 +205,7 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
 | talonscript        | ✅ OK           |                44 |               - |      |
 | tasl               | ✅ OK           |                23 |               - |      |
 | tcl                | ✅ OK           |                34 |               - |      |
-| templ              | ✅ OK           |               680 |               - |      |
+| templ              | ✅ OK           |               682 |               - |      |
 | terraform          | ✅ OK           |                68 |               - |      |
 | tex                | ✅ OK           |               106 |               - |      |
 | toml               | ✅ OK           |                44 |               - |      |
@@ -246,11 +247,10 @@ In some edge cases, it's not guaranteed that the highlighting will be 100% the s
 
 ## Mismatched Languages
 
-Languages that do not throw with the JavaScript RegExp engine, but will produce different results than the WASM engine. Please use with caution.
+Languages that do not throw with the JavaScript RegExp engine, but will produce different results than the WASM engine.
 
-| Language | Highlight Match                                                            | Patterns Parsable | Patterns Failed | Diff |
-| -------- | :------------------------------------------------------------------------- | ----------------: | --------------: | ---: |
-| nginx    | [🚧 Mismatch](https://textmate-grammars-themes.netlify.app/?grammar=nginx) |               378 |               - |  122 |
+| Language | Highlight Match | Patterns Parsable | Patterns Failed | Diff |
+| -------- | :-------------- | ----------------: | --------------: | ---: |
 
 ## Unsupported Languages
 
@@ -262,5 +262,5 @@ Languages that throw with the JavaScript RegExp engine, either because they cont
 | sass       | ✅ OK           |                67 |               2 |      |
 | purescript | ❌ Error        |                72 |               1 |      |
 | csharp     | ❌ Error        |               310 |               3 |  137 |
-| swift      | ❌ Error        |               326 |               3 |    8 |
+| swift      | ❌ Error        |               326 |               3 |      |
 | razor      | ❌ Error        |               957 |               5 |      |
diff --git a/packages/engine-javascript/src/engine-compile.ts b/packages/engine-javascript/src/engine-compile.ts
@@ -38,9 +38,6 @@ export function defaultJavaScriptRegexConstructor(pattern: string, options?: Oni
         asciiWordBoundaries: true,
         // Follow `vscode-oniguruma` which enables this Oniguruma option by default
         captureGroup: true,
-        // Removing `\G` anchors in cases when they're not supported for emulation allows
-        // supporting more grammars, but also allows some mismatches
-        ignoreUnsupportedGAnchors: true,
         // Oniguruma uses depth limit `20`; lowered here to keep regexes shorter and maybe
         // sometimes faster, but can be increased if issues reported due to low limit
         recursionLimit: 5,

diff --git a/packages/langs-precompiled/scripts/__snapshots__/precompile.test.ts.snap b/packages/langs-precompiled/scripts/__snapshots__/precompile.test.ts.snap
@@ -91,7 +91,9 @@ exports[`precompile 1`] = `
           ],
         },
       },
-      end: /(?<=^|\\n(?!$))(?=\\P{space})|(?!)/dgv,
+      end: new EmulatedRegExp("(?<=^|\\\\n(?!$))(?=\\\\P{space})|(?!^)", "dgv", {
+        strategy: "search_start_clip",
+      }),
       patterns: [
         {
           begin: /(?<=^|\\n(?!$))([ ]+)(?! )/dgv,
@@ -110,7 +112,9 @@ exports[`precompile 1`] = `
       beginCaptures: {
         "1": { name: "punctuation.whitespace.comment.leading.yaml" },
       },
-      end: new EmulatedRegExp("", "dgv", { strategy: "not_search_start" }),
+      end: new EmulatedRegExp("(?!^)", "dgv", {
+        strategy: "search_start_clip",
+      }),
       patterns: [
         {
           begin: /#/dgv,
@@ -373,7 +377,9 @@ exports[`precompile 1`] = `
     node: { patterns: [{ include: "#block-node" }] },
     property: {
       begin: /(?=!|&)/dgv,
-      end: new EmulatedRegExp("", "dgv", { strategy: "not_search_start" }),
+      end: new EmulatedRegExp("(?!^)", "dgv", {
+        strategy: "search_start_clip",
+      }),
       name: "meta.property.yaml",
       patterns: [
         {

diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml
@@ -56,7 +56,7 @@ catalog:
   monaco-editor-core: ^0.52.2
   ofetch: ^1.4.1
   ohash: ^1.1.4
-  oniguruma-to-es: ^1.0.0
+  oniguruma-to-es: ^2.0.0
   picocolors: ^1.1.1
   pinia: ^2.3.0
   pnpm: ^9.15.4

diff --git a/scripts/report-engine-js-compat.ts b/scripts/report-engine-js-compat.ts
@@ -237,7 +237,7 @@ async function run() {
     '',
     '## Mismatched Languages',
     '',
-    'Languages that do not throw with the JavaScript RegExp engine, but will produce different results than the WASM engine. Please use with caution.',
+    'Languages that do not throw with the JavaScript RegExp engine, but will produce different results than the WASM engine.',
     '',
     createTable(reportMismatch),
     '',