perf(linter/plugins): collect tokens even when not needed

[overlookmotel]

Unconditionally collect tokens in linter, even if they're not needed (files which are not linted with JS plugins).

This is wasteful in terms of memory, but according to CodSpeed is slightly faster (+0.5%) because it removes the if self.config.tokens() branch from Lexer::finish_next_inner (self.config.tokens() is const-folded to true). The branch costs more than pushing the Token to the Vec<Token>!

But this needs measuring in real world to see what actual perf impact is, so can judge the memory/perf trade-off accurately.

[overlookmotel]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• perf(linter/plugins): collect tokens even when not needed #19970 👈 (View in Graphite)
• perf(linter/plugins): store tokens as a Box<[Token]> #19969
• feat(allocator): add methods for boxed slices ArenaBox<[T]> #19968
• main

---

How to use the Graphite Merge Queue

Add either label to this PR to merge it via the merge queue:

• 0-merge - adds this PR to the back of the merge queue
• hotfix - for urgent hot fixes, skip the queue and merge this PR next

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codspeed-hq]

Merging this PR will not alter performance

✅ 53 untouched benchmarks
⏩ 3 skipped benchmarks¹

---

_{Comparing om/03-03-perf_linter_plugins_collect_tokens_even_when_not_needed (ffead33) with om/03-03-perf_linter_plugins_store_tokens_as_a_box_token_ (c13b437)}

Footnotes

1. 3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[overlookmotel]

I don't think this is worthwhile.