Preparatory changes for macro parsing BFS->DFS by bal-e · Pull Request #158974 · rust-lang/rust

bal-e · 2026-07-08T21:00:05Z

This PR contains some miscellaneous commits I accumulated while working on the BFS->DFS change. Their goal is to simplify the code and clarifying existing behavior. It is a conceptual follow-up to #158577. High-level overview:

Adds context about the current match arm to Tracker through the new Tracker::prepare(), so that the WhichMatcher parameter is available implicitly. In a later PR, this will be used to reference MatcherLocs by index.
Reformulates matching failures to work more like ambiguity errors wrt. Tracker; Tracker::build_failure() (which would build a failure consumed by Tracker::after_arm()) becomes Tracker::failure() which eagerly processes the error. This removes the need for ParseResult::Failure to store any data at all. This relies on the match arm context provided by Tracker::prepare().
There is a subtle edge case involving token::Eof and non-terminal parsing; Parser::nonterminal_may_begin_with() would sometimes return true for token::Eof, even though the non-terminal parse would never be attempted. TtParser did not check bb_mps when handling token::Eof, so non-terminal parses at EOF were being silently dropped. I changed Parser::nonterminal_may_begin_with() to always return false for token::Eof, making this behavior much more visible. I added a test case to make sure meta-variables return the same error uniformly when parsed against EOF.

Contains #158894, which should be merged soon (after which I'll rebase onto main).

Best reviewed commit-by-commit.

r? @nnethercote

rustbot · 2026-07-08T21:00:09Z

The parser was modified, potentially altering the grammar of (stable) Rust
which would be a breaking change.

cc @fmease

rustbot · 2026-07-08T21:00:12Z

Thanks for the pull request, and welcome! The Rust Project is excited to review your changes, and you should hear from @nnethercote (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

petrochenkov · 2026-07-09T11:02:51Z

while working on the BFS->DFS change

Could you tell more, what is the BFS->DFS change and its motivation and goals?

bal-e · 2026-07-09T14:41:32Z

Hey @petrochenkov! I initially got interested in the macro parser when I learned about degenerate cases causing exponential-time (and exponential-space) complexity here. I wrote a blog post about it with more details: https://bal-e.org/blog/2026/oops-cubic-macro/. In short, the macro parser explores ambiguity using a BFS, but it only cares about one result (if there are other results, it only cares about their existence, not the data within; at least on the fast path). A DFS feels simpler here:

It makes it possible to nicely handle the degenerate cases (first by reducing exponential-space usage into linear-space, then making it easy to deduplicate MatcherPos-es to get linear runtime). This is not super important because the degenerate cases rarely occur in practice.
I think it would be faster. With a DFS, you can avoid pushing to and popping off the mps vecs; most of the time you just need a single mp in a local variable and work on that. I suspect LLVM would also have a better time optimizing the control flow, but that's hard to prove.
It makes it possible to share the NamedMatches data between MatcherPos-es. I think it is possible to eliminate the Rcs involved, and later I also want to flatten the data structures to avoid nested heap allocations. This is the biggest motivation.

I'm quite confident the macro parser can be changed to DFS without affecting user behavior (#158894 added a user-visible change, but only for a minor detail in error messages). In any case, I'll write about it more when I get to the main PR. I think these preparatory PRs are worth merging independently of the BFS->DFS change. Happy to answer in more detail too :)

jdonszelmann · 2026-07-09T16:04:39Z

@bors try @rust-timer queue

Preparatory changes for macro parsing BFS->DFS

rust-bors · 2026-07-09T18:12:09Z

☀️ Try build successful (CI)
Build commit: 71875d2 (71875d228f62083c0fb1d3efb367bcae29b400d2)
Base parent: 561ea2b (561ea2bfa3022226082e2db39d6ccb241e9e34bc)

rust-timer · 2026-07-09T19:30:04Z

Finished benchmarking commit (71875d2): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.3%]	8
Regressions ❌ (secondary)	0.3%	[0.2%, 0.3%]	11
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.2%]	10
Improvements ✅ (secondary)	-4.4%	[-6.7%, -0.1%]	10
All ❌✅ (primary)	-0.0%	[-0.3%, 0.3%]	18

Max RSS (memory usage)

Results (primary 0.8%, secondary 4.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	3.4%	[3.4%, 3.4%]	1
Regressions ❌ (secondary)	4.8%	[2.9%, 6.7%]	2
Improvements ✅ (primary)	-1.8%	[-1.8%, -1.8%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.8%	[-1.8%, 3.4%]	2

Cycles

Results (primary -2.6%, secondary -6.9%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.0%	[3.0%, 3.0%]	1
Improvements ✅ (primary)	-2.6%	[-2.6%, -2.6%]	1
Improvements ✅ (secondary)	-8.1%	[-13.2%, -2.7%]	8
All ❌✅ (primary)	-2.6%	[-2.6%, -2.6%]	1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.147s -> 487.834s (-0.27%)
Artifact size: 388.47 MiB -> 388.84 MiB (0.10%)

In the next commit, the work done in `Tracker::after_arm()` upon a `Failure` will be moved into `Tracker::build_failure()`. For this to work, the `WhichMatcher` parameter to `after_arm()` needs to be available to `build_failure()`. Rather than threading it through `TtParser`, this commit stores it in `CollectTrackerAndEmitter` and keeps it updated using a new `Tracker::prepare()` method. `tests/ui/macros` pass.

Rather than preparing a failure using `Tracker::build_failure()` and consuming it from `Tracker::after_arm()`, the parser now calls `Tracker::failure()` the moment a failure is detected, and this will eagerly perform all the necessary processing. This makes the contents of `Failure` redundant, so they can be removed in the next commit.

Now that everything is processed in `Tracker::failure()`, this data does not need to be propagated. This also removes some generics.

The appropriate token (span) and the diagnostic message are now computed in `diagnostics.rs`. Following this commit, _all_ diagnostic messages have been moved from `macro_parser.rs` to `diagnostics.rs`. Yay!

It is being used by `Parser::nonterminal_may_begin_with()`.

This commit deliberately causes panics. See the next commit for more details and a fix.

Until now, `:tt`, `:item`, and `:stmt` metavars could be attempted against `token::Eof`. *HOWEVER*, when a `token::Eof` is processed, `parse_tt_inner()` ignores `bb_mps`, not even checking for ambiguity. It's as if those `bb_mps` never existed; as if `nonterminal_may_begin_with()` returned `false` for `Eof`. This commit makes that behavior more explicit. This does not change user visible behavior. `tests/ui/macros` pass.

The previous commit fixed a case where some meta-variables were being treated as if they *could* match `token::Eof` and then silently ignored. This commit adds a test to ensure meta-variables continue to have a consistent response to an `Eof` (in theory, they could vary between fatal non-terminal parsing errors and non-fatal matching failures).

rust-bors · 2026-07-11T01:30:12Z

💔 Test for f402c2a failed: CI. Failed job:

auto - test-various (web logs, enhanced plaintext logs)

JonathanBrouwer · 2026-07-11T09:58:20Z

@bors retry

@nnethercote

Preparatory changes for macro parsing BFS->DFS This PR contains some miscellaneous commits I accumulated while working on the BFS->DFS change. Their goal is to simplify the code and clarifying existing behavior. It is a conceptual follow-up to #158577. High-level overview: - Adds context about the current match arm to `Tracker` through the new `Tracker::prepare()`, so that the `WhichMatcher` parameter is available implicitly. In a later PR, this will be used to reference `MatcherLoc`s by index. - Reformulates matching failures to work more like ambiguity errors wrt. `Tracker`; `Tracker::build_failure()` (which would build a failure consumed by `Tracker::after_arm()`) becomes `Tracker::failure()` which eagerly processes the error. This removes the need for `ParseResult::Failure` to store any data at all. This relies on the match arm context provided by `Tracker::prepare()`. - There is a subtle edge case involving `token::Eof` and non-terminal parsing; `Parser::nonterminal_may_begin_with()` would sometimes return `true` for `token::Eof`, even though the non-terminal parse would never be attempted. `TtParser` did not check `bb_mps` when handling `token::Eof`, so non-terminal parses at EOF were being silently dropped. I changed `Parser::nonterminal_may_begin_with()` to always return `false` for `token::Eof`, making this behavior much more visible. I added a test case to make sure meta-variables return the same error uniformly when parsed against EOF. Contains #158894, which should be merged soon (after which I'll rebase onto `main`). Best reviewed commit-by-commit. r? @nnethercote

rust-bors · 2026-07-11T23:26:31Z

💔 Test for ea14f2e failed: CI. Failed job:

auto - i686-gnu-nopt-1 (web logs, enhanced plaintext logs)

rust-log-analyzer · 2026-07-11T23:26:33Z

The job i686-gnu-nopt-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

JonathanBrouwer · 2026-07-12T06:12:22Z

@bors retry
@bors try jobs=i686-gnu-nopt-1

Preparatory changes for macro parsing BFS->DFS try-job: i686-gnu-nopt-1

rust-bors · 2026-07-12T08:44:24Z

☀️ Try build successful (CI)
Build commit: 7d0def3 (7d0def3f16a2f942d6ce0d8b34ddda7290b3fe4c)
Base parent: be8e824 (be8e82435eb04fbe75ed5286b52735366e160bed)

JonathanBrouwer · 2026-07-12T15:13:08Z

@bors p=6
Scheduling this before the rollup, since the rollup is still small

rust-bors · 2026-07-12T19:32:40Z

☀️ Test successful - CI
Approved by: nnethercote
Duration: 3h 19m 39s
Pushing 77cf889 to main...

github-actions · 2026-07-12T19:36:11Z

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing f0b782b (parent) -> 77cf889 (this PR)

Test differences

Show 5 test diffs

Stage 1

[ui] tests/ui/macros/metavar-at-eof.rs: [missing] -> pass (J1)
[ui (polonius)] tests/ui/macros/metavar-at-eof.rs: [missing] -> pass (J2)

Stage 2

[ui] tests/ui/macros/metavar-at-eof.rs: [missing] -> pass (J0)

Additionally, 2 doctest diffs were found. These are ignored, as they are noisy.

Job group index

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 77cf889bc178ddb44d6a1c78e5a820b5abb31d8d --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

x86_64-msvc-ext3: 59m 54s -> 1h 50m (+83.6%)
x86_64-gnu-gcc-core-tests: 8m 5s -> 14m 9s (+75.1%)
dist-x86_64-solaris: 58m 32s -> 1h 39m (+70.8%)
armhf-gnu: 59m 32s -> 1h 35m (+60.7%)
x86_64-gnu-tools: 44m 56s -> 1h 7m (+49.7%)
x86_64-gnu-gcc: 49m 43s -> 1h 13m (+48.2%)
dist-aarch64-linux: 1h 45m -> 2h 34m (+46.2%)
x86_64-msvc-2: 1h 51m -> 2h 25m (+30.5%)
x86_64-gnu: 2h 24m -> 1h 40m (-30.4%)
i686-gnu-1: 1h 46m -> 2h 17m (+29.3%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

rust-timer · 2026-07-12T21:46:02Z

Finished benchmarking commit (77cf889): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Our benchmarks found a performance regression caused by this PR.
This might be an actual regression, but it can also be just noise.

Next Steps:

If the regression was expected or you think it can be justified,
please write a comment with sufficient written justification, and add
@rustbot label: +perf-regression-triaged to it, to mark the regression as triaged.
If you think that you know of a way to resolve the regression, try to create
a new PR with a fix for the regression.
If you do not understand the regression or you think that it is just noise,
you can ask the @rust-lang/wg-compiler-performance working group for help (members of this group
were already notified of this PR).

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.2%, 0.3%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	2
Improvements ✅ (secondary)	-3.8%	[-5.2%, -2.1%]	9
All ❌✅ (primary)	-0.1%	[-0.4%, 0.3%]	4

Max RSS (memory usage)

Results (primary 2.0%, secondary 1.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.0%	[1.9%, 2.0%]	2
Regressions ❌ (secondary)	3.6%	[2.0%, 8.9%]	5
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.9%	[-3.9%, -1.0%]	3
All ❌✅ (primary)	2.0%	[1.9%, 2.0%]	2

Cycles

Results (secondary -0.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.8%	[3.5%, 4.2%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.6%	[-2.9%, -2.3%]	3
All ❌✅ (primary)	-	-	0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 489.629s -> 489.838s (0.04%)
Artifact size: 391.96 MiB -> 389.25 MiB (-0.69%)

@nnethercote

Eagerly check for ambiguity in macro parsing This PR implements an important step leading up to the BFS->DFS change: it makes ambiguity detection (as occurs when parsing meta-variables and reaching EOF) eager. Rather than accumulating the `bb_mps` and `eof_mps` lists, then checking that a single valid parse exists, this PR introduces a `check_for_ambiguity()` method that gets called as soon as the relevant `MatcherLoc` is observed. `check_for_ambiguity()` immediately drains `cur_mps` instead of waiting for the outer loop to do so, and evaluates the mps within a "checking for ambiguity" context. This change was complicated by the fact that `bb_mps` was relied on for error messages; it was passed along to `Tracker::ambiguity()`. The same is true for `next_mps`. The first few commits of this PR (after pending rebases) add state to `Tracker` so that it can compute the relevant diagnostic information independently of `bb_mps`. This PR complicates the control flow around `parse_tt_inner()` in subtle ways. I'm not super happy about it, but I think it will become easier to understand again once a DFS approach is implemented. ~This PR contains #158894 and #158974. I'll rebase on top of `main` once they're both merged.~ Best reviewed commit-by-commit. r? @nnethercote

@nnethercote

Eagerly check for ambiguity in macro parsing This PR implements an important step leading up to the BFS->DFS change: it makes ambiguity detection (as occurs when parsing meta-variables and reaching EOF) eager. Rather than accumulating the `bb_mps` and `eof_mps` lists, then checking that a single valid parse exists, this PR introduces a `check_for_ambiguity()` method that gets called as soon as the relevant `MatcherLoc` is observed. `check_for_ambiguity()` immediately drains `cur_mps` instead of waiting for the outer loop to do so, and evaluates the mps within a "checking for ambiguity" context. This change was complicated by the fact that `bb_mps` was relied on for error messages; it was passed along to `Tracker::ambiguity()`. The same is true for `next_mps`. The first few commits of this PR (after pending rebases) add state to `Tracker` so that it can compute the relevant diagnostic information independently of `bb_mps`. This PR complicates the control flow around `parse_tt_inner()` in subtle ways. I'm not super happy about it, but I think it will become easier to understand again once a DFS approach is implemented. ~This PR contains rust-lang/rust#158894 and rust-lang/rust#158974. I'll rebase on top of `main` once they're both merged.~ Best reviewed commit-by-commit. r? @nnethercote

@nnethercote

Eagerly check for ambiguity in macro parsing This PR implements an important step leading up to the BFS->DFS change: it makes ambiguity detection (as occurs when parsing meta-variables and reaching EOF) eager. Rather than accumulating the `bb_mps` and `eof_mps` lists, then checking that a single valid parse exists, this PR introduces a `check_for_ambiguity()` method that gets called as soon as the relevant `MatcherLoc` is observed. `check_for_ambiguity()` immediately drains `cur_mps` instead of waiting for the outer loop to do so, and evaluates the mps within a "checking for ambiguity" context. This change was complicated by the fact that `bb_mps` was relied on for error messages; it was passed along to `Tracker::ambiguity()`. The same is true for `next_mps`. The first few commits of this PR (after pending rebases) add state to `Tracker` so that it can compute the relevant diagnostic information independently of `bb_mps`. This PR complicates the control flow around `parse_tt_inner()` in subtle ways. I'm not super happy about it, but I think it will become easier to understand again once a DFS approach is implemented. ~This PR contains rust-lang/rust#158894 and rust-lang/rust#158974. I'll rebase on top of `main` once they're both merged.~ Best reviewed commit-by-commit. r? @nnethercote

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jul 8, 2026

rustbot assigned nnethercote Jul 8, 2026

bal-e mentioned this pull request Jul 8, 2026

Eagerly check for ambiguity in macro parsing #158976

Merged

petrochenkov self-assigned this Jul 9, 2026

petrochenkov removed their assignment Jul 9, 2026

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Jul 9, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jul 9, 2026

Auto merge of #158974 - bal-e:macro-parsing-dfs-prep, r=<try>

71875d2

Preparatory changes for macro parsing BFS->DFS

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Jul 9, 2026

arya dradjica added 10 commits July 10, 2026 08:17

Minor fixups related to ambiguity errors

70890ca

Remove data from ParseResult::Failure and remove Tracker::Failure

b225a7e

Now that everything is processed in `Tracker::failure()`, this data does not need to be propagated. This also removes some generics.

Only pass a Parser to Tracker::failure()

3153216

The appropriate token (span) and the diagnostic message are now computed in `diagnostics.rs`. Following this commit, _all_ diagnostic messages have been moved from `macro_parser.rs` to `diagnostics.rs`. Yay!

Simplify failure message computation

247b0ec

Remove incorrect comment on can_begin_pattern()

baf59c2

It is being used by `Parser::nonterminal_may_begin_with()`.

Ensure that no mps are left over on EOF

05ffc92

This commit deliberately causes panics. See the next commit for more details and a fix.

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 11, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 11, 2026

rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 12, 2026

This comment has been minimized.

Sign in to view

rust-bors Bot pushed a commit that referenced this pull request Jul 12, 2026

Auto merge of #158974 - bal-e:macro-parsing-dfs-prep, r=<try>

7d0def3

Preparatory changes for macro parsing BFS->DFS try-job: i686-gnu-nopt-1

This comment has been minimized.

Sign in to view

rust-bors Bot added merged-by-bors This PR was explicitly merged by bors. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 12, 2026

rust-bors Bot merged commit 77cf889 into rust-lang:main Jul 12, 2026
15 checks passed

rustbot added this to the 1.99.0 milestone Jul 12, 2026

bal-e deleted the macro-parsing-dfs-prep branch July 12, 2026 20:36

bal-e mentioned this pull request Jul 24, 2026

Use depth-first traversal for macro parsing #159808

Open

Uh oh!

Uh oh!

Conversation

bal-e commented Jul 8, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 8, 2026

Uh oh!

rustbot commented Jul 8, 2026

Uh oh!

petrochenkov commented Jul 9, 2026

Uh oh!

bal-e commented Jul 9, 2026

Uh oh!

jdonszelmann commented Jul 9, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors Bot commented Jul 9, 2026

Uh oh!

This comment has been minimized.

rust-timer commented Jul 9, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

rust-bors Bot commented Jul 11, 2026

Uh oh!

JonathanBrouwer commented Jul 11, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jul 11, 2026

Uh oh!

rust-log-analyzer commented Jul 11, 2026

Uh oh!

JonathanBrouwer commented Jul 12, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jul 12, 2026

Uh oh!

JonathanBrouwer commented Jul 12, 2026

Uh oh!

This comment has been minimized.

rust-bors Bot commented Jul 12, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 12, 2026

Test differences

Stage 1

Stage 2

Job duration changes

Uh oh!

rust-timer commented Jul 12, 2026

Overall result: ❌✅ regressions and improvements - please read:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bal-e commented Jul 8, 2026 •

edited by rustbot

Loading