fix(js_lexer): regex escape char

[Sec-ant]

Summary

After the lexer reads an escape char \ in a regular expression, it should advance a byte or a char, so that valid regexes like /\’/ won't fail.

Fixes #1941.

Test Plan

A test case is added in crates/biome_js_parser/test_data/inline/ok/literals.js

[netlify]

✅ Deploy Preview for biomejs ready!

Name	Link
🔨 Latest commit	8964131
🔍 Latest deploy log	https://app.netlify.com/sites/biomejs/deploys/65ef1ba04cd60b0008585601
😎 Deploy Preview	https://deploy-preview-2044--biomejs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	1 paths audited Performance: 100 (no change from production) Accessibility: 97 (no change from production) Best Practices: 100 (no change from production) SEO: 93 (no change from production) PWA: - View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[arendjr]

Nice! I would appreciate if we can add a test case for this, but the fix makes sense 👍

[codspeed-hq]

CodSpeed Performance Report

Merging #2044 will degrade performances by 18.31%

_{Comparing Sec-ant:fix-regex-lexer (8964131) with main (57fa936)}

Summary

⚡ 1 improvements
❌ 2 regressions
✅ 90 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	Sec-ant:fix-regex-lexer	Change
❌	router.ts[cached]	10.8 ms	11.6 ms	-6.72%
❌	big5-added.json[cached]	2.2 ms	2.7 ms	-18.31%
⚡	db.json[cached]	77.2 ms	71.8 ms	+7.63%

[arendjr]

Lexer test cases can be added here, btw: https://github.com/biomejs/biome/blob/main/crates/biome_js_parser/src/lexer/tests.rs

[Conaclos]

Could you add a changelog entry in CHANGELOG.md? I just committed the Unreleased section on main. You could rebase your PR on main :)

EDIT: you can find some guidelines in our contributing guide.

[Sec-ant]

Lexer test cases can be added here, btw: https://github.com/biomejs/biome/blob/main/crates/biome_js_parser/src/lexer/tests.rs

Yes I was hoping to add a test here, something like this:

#[test]
fn issue_1941() {
    assert_lex! {
        r#"/\’/"#,
        JS_REGEX_LITERAL:6
    }
}

But the lexer needs some context to read this as a regex, or it will produce SLASH:1 ERROR:1 ERROR:3 SLASH:1. I don't know how should I give it the proper context?

[arendjr]

I think r#"let a = /\’/"# should do the trick.

[Sec-ant]

I think r#"let a = /\’/"# should do the trick.

Still slash and error token. Did I do something wrong?

[
    (
        LET_KW,
        0..3,
    ),
    (
        WHITESPACE,
        3..4,
    ),
    (
        IDENT,
        4..5,
    ),
    (
        WHITESPACE,
        5..6,
    ),
    (
        EQ,
        6..7,
    ),
    (
        WHITESPACE,
        7..8,
    ),
    (
        SLASH,
        8..9,
    ),
    (
        ERROR_TOKEN,
        9..10,
    ),
    (
        ERROR_TOKEN,
        10..13,
    ),
    (
        SLASH,
        13..14,
    ),
]

[arendjr]

No, it's not your fault. It looks like the lexer cannot determine the presence of regexes on its own. It only happens after the parser tells it to "re-lex" certain nodes. So instead, I think you should add your test case here: https://github.com/biomejs/biome/blob/main/crates/biome_js_parser/test_data/inline/ok/literals.js

[Sec-ant]

@arendjr Yes here it works, thanks!

[Sec-ant]

Could you add a changelog entry in CHANGELOG.md? I just committed the Unreleased section on main. You could rebase your PR on main :)

EDIT: you can find some guidelines in our contributing guide.

@Conaclos Changelog entry added. Should I add changelogs for other PRs (post v1.60) as well? Some are already merged.

[arendjr]

Looks good!

[arendjr]

Should I add changelogs for other PRs (post v1.60) as well? Some are already merged.

Might be better to do that in a separate PR, I think.

[arendjr]

Could you run the codegen one more time to update the changelog for the website?

[Sec-ant]

There're over 100 files modified after just gen. One is the changelog entry I added, but others seem to be introduced from the main branch when I rebased. I splitted them into two commits.

[arendjr]

Thanks, I guess something was pushed to main without running the codegen. Still waiting for the CI, but I think it will pass 🤞

[arendjr]

Thanks for the patch!