Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(parser): implement NthToken for T #2727

Merged
merged 2 commits into from
May 7, 2024
Merged

refactor(parser): implement NthToken for T #2727

merged 2 commits into from
May 7, 2024

Conversation

denbezrukov
Copy link
Contributor

@denbezrukov denbezrukov commented May 5, 2024

Summary

We have code duplication around the non-trivia lookahead in our parser. The MR attempts to extract the common parts into the parser crate. Basically, we have a non_trivia_lookahead vector for each token source. We can try to move them inside the current lookahead in the buffered lexer, because we are interested only in non-trivia lookahead.

Test Plan

cargo test

@github-actions github-actions bot added A-Core Area: core A-Parser Area: parser L-JavaScript Language: JavaScript and super languages L-CSS Language: CSS L-JSON Language: JSON and super languages labels May 5, 2024
Copy link
Contributor

github-actions bot commented May 5, 2024

Parser conformance results on

js/262

Test result main count This PR count Difference
Total 49701 49701 0
Passed 48721 48721 0
Failed 980 980 0
Panics 0 0 0
Coverage 98.03% 98.03% 0.00%

jsx/babel

Test result main count This PR count Difference
Total 40 40 0
Passed 37 37 0
Failed 3 3 0
Panics 0 0 0
Coverage 92.50% 92.50% 0.00%

symbols/microsoft

Test result main count This PR count Difference
Total 6322 6322 0
Passed 2036 2036 0
Failed 4286 4286 0
Panics 0 0 0
Coverage 32.20% 32.20% 0.00%

ts/babel

Test result main count This PR count Difference
Total 662 662 0
Passed 593 593 0
Failed 69 69 0
Panics 0 0 0
Coverage 89.58% 89.58% 0.00%

ts/microsoft

Test result main count This PR count Difference
Total 17646 17646 0
Passed 13439 13439 0
Failed 4205 4205 0
Panics 2 2 0
Coverage 76.16% 76.16% 0.00%

Copy link

codspeed-hq bot commented May 5, 2024

CodSpeed Performance Report

Merging #2727 will not alter performance

Comparing chore/nth (1d55e30) with main (c269016)

Summary

✅ 85 untouched benchmarks

@denbezrukov denbezrukov changed the title refactor(parser): Implement NthToken for T refactor(parser): implement NthToken for T May 6, 2024
@denbezrukov denbezrukov marked this pull request as ready for review May 6, 2024 19:02
@denbezrukov denbezrukov requested a review from ematipico May 6, 2024 19:02
Copy link
Member

@ematipico ematipico left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, you managed to kill the lifetime! (To reduce it during lexing)

I did try to do that some time ago but then stopped.

Great refactor! Is there some documentation (contribution guide) we need to update?

/// Creates a new [BufferedLexer] wrapping the passed in [Lexer].
pub fn new(lexer: Lex) -> Self {
Self {
inner: lexer,
current: None,
lookahead: VecDeque::new(),
lookahead: Lookahead::with_capacity(5),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 5?

Copy link
Contributor Author

@denbezrukov denbezrukov May 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh,
I wanted to look at the performance tests, but it seems there is no effect,
Do you think it's better to use just new?
5 is a the longest lookahed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, if there's no change in performance, maybe we should stick to new

@denbezrukov denbezrukov merged commit 01c440d into main May 7, 2024
16 checks passed
@denbezrukov denbezrukov deleted the chore/nth branch May 7, 2024 04:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Core Area: core A-Parser Area: parser L-CSS Language: CSS L-JavaScript Language: JavaScript and super languages L-JSON Language: JSON and super languages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants