This repository has been archived by the owner on Aug 31, 2023. It is now read-only.
perf(rome_js_formatter): Reduce the String allocations for Tokens
#2462
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ematipico
reviewed
Apr 19, 2022
| @@ -1125,6 +1127,13 @@ pub enum Token { | |||
| // The position of the dynamic token in the unformatted source code | |||
| source_position: TextSize, | |||
| }, | |||
| // A token that is taken 1:1 from the source code | |||
There was a problem hiding this comment.
Suggested change
| // A token that is taken 1:1 from the source code | |
| /// A token that is taken 1:1 from the source code |
What does "taken" mean exactly here? Computed? Extracted?
There was a problem hiding this comment.
I can try to improve the existing documentation. It was neither computed nor extracted. It just means it's a text that we keep as is, the formatting doesn't change it.
There was a problem hiding this comment.
Maybe "borrowed" could be a more familiar wording, although in Rust it implies some lifetime semantics that are not present here
leops
reviewed
Apr 19, 2022
| @@ -1125,6 +1127,13 @@ pub enum Token { | |||
| // The position of the dynamic token in the unformatted source code | |||
| source_position: TextSize, | |||
| }, | |||
| // A token that is taken 1:1 from the source code | |||
There was a problem hiding this comment.
Maybe "borrowed" could be a more familiar wording, although in Rust it implies some lifetime semantics that are not present here
Deploying with
|
| Latest commit: |
a93522b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://57a4bc55.tools-8rn.pages.dev |
ematipico
approved these changes
Apr 20, 2022
leops
approved these changes
Apr 20, 2022
This PR reduces the amount of `String` allocation necessary for `FormatElement::Token`s by making use of the observation that most tokens match the text of a `SyntaxToken`. For example, identifiers, or punctuation tokens are kept by the formatter as is. This is even true for string literal tokens if they already use the right quotes. The way this is implemented is by introducing a new `SyntaxTokenText` that is `Send + Sync` and allows referencing a slice in a `SyntaxToken` without worrying about the `&str`'s lifetime. The PR further extends `FormatElement::Token` to make use of this new introduced `SyntaxTokenText`. This change reduces overall memory consumption and improves performance: ``` group format-element token ----- -------------- ----- formatter/checker.ts 1.04 250.2±5.03ms 10.4 MB/sec 1.00 239.5±1.76ms 10.9 MB/sec formatter/compiler.js 1.07 145.6±1.30ms 7.2 MB/sec 1.00 136.5±1.43ms 7.7 MB/sec formatter/d3.min.js 1.07 117.4±3.70ms 2.2 MB/sec 1.00 109.6±1.24ms 2.4 MB/sec formatter/dojo.js 1.03 7.4±0.15ms 9.2 MB/sec 1.00 7.2±0.03ms 9.5 MB/sec formatter/ios.d.ts 1.05 181.2±1.95ms 10.3 MB/sec 1.00 172.8±2.23ms 10.8 MB/sec formatter/jquery.min.js 1.02 29.1±0.55ms 2.8 MB/sec 1.00 28.5±0.07ms 2.9 MB/sec formatter/math.js 1.05 233.1±4.69ms 2.8 MB/sec 1.00 222.8±1.79ms 2.9 MB/sec formatter/parser.ts 1.03 5.3±0.15ms 9.2 MB/sec 1.00 5.1±0.01ms 9.5 MB/sec formatter/pixi.min.js 1.10 131.0±7.11ms 3.3 MB/sec 1.00 119.3±2.12ms 3.7 MB/sec formatter/react-dom.production.min.js 1.07 37.0±0.82ms 3.1 MB/sec 1.00 34.5±0.21ms 3.3 MB/sec formatter/react.production.min.js 1.08 1825.1±57.85µs 3.4 MB/sec 1.00 1683.8±30.49µs 3.7 MB/sec formatter/router.ts 1.02 3.7±0.09ms 16.2 MB/sec 1.00 3.6±0.01ms 16.6 MB/sec formatter/tex-chtml-full.js 1.05 288.3±5.19ms 3.2 MB/sec 1.00 273.4±1.29ms 3.3 MB/sec formatter/three.min.js 1.11 155.7±3.79ms 3.8 MB/sec 1.00 139.7±1.76ms 4.2 MB/sec formatter/typescript.js 1.04 945.2±6.64ms 10.1 MB/sec 1.00 909.3±7.16ms 10.4 MB/sec formatter/vue.global.prod.js 1.07 49.1±1.49ms 2.5 MB/sec 1.00 45.8±0.20ms 2.6 MB/sec ```
MichaReiser
force-pushed
the
perf/syntax-token-slice
branch
from
245c0ea
to
a93522b
Compare
Parser conformance results on ubuntu-latestjs/262
jsx/babel
ts/babel
ts/microsoft
|
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reduces the amount of
Stringallocation necessary forFormatElement::Tokens bymaking use of the observation that most tokens match the text of a
SyntaxToken. For example,identifiers, or punctuation tokens are kept by the formatter as is. This is even true for string literal tokens if they already use the right quotes.
The way this is implemented is by introducing a new
SyntaxTokenTextthat isSend + Syncand allows referencing a slice in aSyntaxTokenwithout worrying about the&str's lifetime. The PR further extendsFormatElement::Tokento make use of this new introducedSyntaxTokenText.This change reduces overall memory consumption and improves performance: