fix(parser): store lone surrogates as escape sequence

[overlookmotel]

Closes #3526.

Fully parse strings containing lone surrogates and encode the string in value.

Encoding schema is to encode a lone surrogate as the lossy replacement character, followed by the code point in hex. i.e. "\uD800" in source code would be encoded as \u{FFFD}d800 in value. The lossy replacement character itself is encoded as \u{FFFD}fffd.

There's nothing special about the lossy replacement character. Just had to choose some valid Unicode character to be the escape marker, and that seemed like a reasonable choice of a character which is likely to be rare in real-world code.

All the ESTree-Test262 test cases related to lone surrogates now pass.

WIP. A bit of tidying up to do yet.

[overlookmotel]

• perf(codegen): speed up printing StringLiterals #10046
• ci(codegen): add benchmark for minified printing #10109
• fix(codegen): do not escape $ in strings unless using backtick as quote #10103
• fix(codegen): prevent arithmetic overflow calculating quote for StringLiterals #10102
• perf(codegen): faster printing quotes #10080
• refactor(codegen): print string literals containing lone surrogates without reference to raw #10044
• fix(parser): store lone surrogates as escape sequence #10041 👈 (View in Graphite)
• test(minifier): update cargo minsize snapshot #10105
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #10041 will not alter performance

_{Comparing 03-25-fix_parser_store_lone_surrogates_as_escape_sequence (f0e1510) with 03-28-test_minifier_update_cargo_minsize_snapshot (fe8625d)}

Summary

✅ 33 untouched benchmarks

[Boshen]

cc @andreubotella for a quick review.

[andreubotella]

cc @andreubotella for a quick review.

IIUC, if lone_surrogates is false, then any replacement characters in the value are true replacement characters; and if true, then the value string would need decoding, right? I think we can work with that in Nova. (cc @aapoalas)

That said, I worry that other users of oxc_ast might not be aware of this change, and might use value without checking for lone_surrogate. You could say that no AsRef<str> would accurately represent the JS string, but I'd say that decoding any lone surrogate lossily to the replacement character (as String::from_utf16_lossy does) would be a lot more useful for most use cases. If the extra memory use is not an issue, I'd suggest that value should be this lossy decode, and lone_surrogate_string would be an Option<Atom<'a>> that is only Some if there are lone surrogates.

[overlookmotel]

Something is wrong. This PR should make no difference to result of cargo minsize, because codegen is only using the value of raw when lone_surrogates flag is set (prior to #10044). I'll investigate.

[overlookmotel]

Something is wrong. This PR should make no difference to result of cargo minsize, because codegen is only using the value of raw when lone_surrogates flag is set (prior to #10044). I'll investigate.

Nope. Snapshot was just out of date.

I just need to add some tests, and this should be good to go.

[graphite-app]

Merge activity

• Mar 29, 8:29 AM EDT: Boshen added this pull request to the Graphite merge queue.
• Mar 29, 9:04 AM EDT: A user merged this pull request with the Graphite merge queue.

[overlookmotel]

@overlookmotel Note to self: Add some tests. Especially for strings containing both lone surrogates and lossy replacement characters.

[overlookmotel]

Tests added in #10175.