perf(ast_tools): parse files in parallel#20578
Merged
graphite-app[bot] merged 1 commit intomainfrom Mar 20, 2026
Merged
Conversation
Member
Author
This was referenced Mar 20, 2026
github-actions
bot
added
the
C-performance
This was referenced Mar 20, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Speeds up oxc_ast_tools schema construction by parallelizing the phase-1 file parsing across crates/files, while keeping deterministic ordering for generated TypeId/MetaId indexing.
Changes:
- Parse/load Rust files in parallel via
rayon::par_iter()and then merge results sequentially to preserve deterministic ordering. - Refactor phase-1 collection to use
FxIndexSet+IndexVecinstead ofFxIndexMapfor skeleton storage and name indexing. - Adjust
synfeature flags and introduce anunsafe Sendwrapper to allow transferringsynASTs between threads.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tasks/ast_tools/src/parse/mod.rs | Parallelizes file loading and introduces the AssertSend wrapper for syn AST transfer. |
| tasks/ast_tools/src/parse/load.rs | Changes load_file to return per-file skeleton results instead of mutating shared maps. |
| tasks/ast_tools/src/parse/parse.rs | Updates parse() signature to accept sets/vectors aligned with new phase-1 data structures. |
| tasks/ast_tools/Cargo.toml | Updates syn feature selection (drops proc-macro explicitly). |
overlookmotel
force-pushed
the
om/02-14-perf_ast_tools_parse_files_in_parallel
branch
from
93f5bde to
c66b283
Compare
Contributor
Merge activity
|
#20577 made `ast_tools` search for and parse all files in all crates that depend on `oxc_ast_macros`. The one downside of this change is that it makes running `ast_tools` slower. Speed it up again, by parsing files in parallel, using `rayon`. There is one annoyance. Due to feature unification, `syn` crate gets compiled with `proc-macro` feature enabled, which makes `syn`s ASTs not `Send`. We don't use the AST in ways which can produce data races (and we can't, because `ast_tools` isn't a proc macro), so it is safe in practice to send ASTs across threads here. So we have to hack it to make it compile (see comment in `tasks/ast_tools/src/parse/mod.rs`).
graphite-app
bot
force-pushed
the
om/02-14-refactor_ast_tools_search_all_crates_which_depend_on_oxc_ast_macros_
branch
from
98a1757 to
1be1ebe
Compare
graphite-app
bot
force-pushed
the
om/02-14-perf_ast_tools_parse_files_in_parallel
branch
from
c66b283 to
82a136f
Compare
Base automatically changed from
om/02-14-refactor_ast_tools_search_all_crates_which_depend_on_oxc_ast_macros_
to
main
March 20, 2026 22:22
graphite-app
bot
deleted the
om/02-14-perf_ast_tools_parse_files_in_parallel
branch
costajohnt
pushed a commit
to costajohnt/oxc
that referenced
this pull request
Mar 22, 2026
oxc-project#20577 made `ast_tools` search for and parse all files in all crates that depend on `oxc_ast_macros`. The one downside of this change is that it makes running `ast_tools` slower. Speed it up again, by parsing files in parallel, using `rayon`. There is one annoyance. Due to feature unification, `syn` crate gets compiled with `proc-macro` feature enabled, which makes `syn`s ASTs not `Send`. We don't use the AST in ways which can produce data races (and we can't, because `ast_tools` isn't a proc macro), so it is safe in practice to send ASTs across threads here. So we have to hack it to make it compile (see comment in `tasks/ast_tools/src/parse/mod.rs`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#20577 made
ast_toolssearch for and parse all files in all crates that depend onoxc_ast_macros.The one downside of this change is that it makes running
ast_toolsslower.Speed it up again, by parsing files in parallel, using
rayon.There is one annoyance. Due to feature unification,
syncrate gets compiled withproc-macrofeature enabled, which makessyns ASTs notSend. We don't use the AST in ways which can produce data races (and we can't, becauseast_toolsisn't a proc macro), so it is safe in practice to send ASTs across threads here. So we have to hack it to make it compile (see comment intasks/ast_tools/src/parse/mod.rs).