Unstable book options parser by mehdiakiki · Pull Request #154070 · rust-lang/rust

mehdiakiki · 2026-03-19T02:04:57Z

Parses the options! macro in compiler/rustc_session/src/options.rs directly to extract the unstable (-Z) compiler flag names and descriptions to generate documentation for the unstable book.

I took notice from the previous attempt which ran rustc -Zhelp and parsed the output and used this approach that reads the source directly.

Used claude for the tedious char by char parsing parts but verified the code, I hope that's ok!

rustbot · 2026-03-19T02:05:01Z

Some changes occurred in compiler/rustc_attr_parsing

cc @jdonszelmann, @JonathanBrouwer

Some changes occurred in compiler/rustc_passes/src/check_attr.rs

cc @jdonszelmann, @JonathanBrouwer

rustbot · 2026-03-19T02:05:03Z

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

Owners of files modified in this PR: compiler
compiler expanded to 69 candidates
Random selection from 15 candidates

jyn514 · 2026-03-19T04:55:50Z

src/tools/unstable-book-gen/src/main.rs

+    next_idx: usize,
+}
+
+fn parse_compiler_flags(options_rs: &str, options_path: &Path) -> Features {


is there any way you can use syn here instead of open-coding it?

alternatively, maybe we could move to the options to a separate file, use include! in the existing rust file, and then require them to be in a simpler format that's easier for syn to parse?

Hi @jyn514 ! Currently I am parsing the whole file as a string, I will have a look at syn and report back!

Parsing is much more straightforward with syn. Here’s my first attempt and I’d appreciate any feedback.

petrochenkov · 2026-03-19T09:27:24Z

r? rustdoc I guess

jyn514 · 2026-03-19T11:26:30Z

r? bootstrap

rustbot · 2026-03-21T16:07:14Z

This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

clubby789 · 2026-03-23T13:00:09Z

It seems a bit unfortunate that we're essentially implementing this macro twice (once as a decl macro, once as a Syn parser) but it's probably worth the benefit. Can we update the options! macro definition to mention that this parser needs to be kept up to date?

clubby789 · 2026-03-23T13:00:30Z

src/tools/unstable-book-gen/src/main.rs

+            return Err(syn::Error::new(
+                name.span(),
+                format!(
+                    "unexpected field count for option `{name}`: expected 4 or 5, found {}",


Suggested change

"unexpected field count for option `{name}`: expected 4 or 5, found {}",

"unexpected field count for option `{name}`: expected {REQUIRED_FIELDS} or {OPTIONAL_FIELDS}, found {}",

@clubby789 sure, will do by tomorrow!

rustbot · 2026-03-25T01:50:10Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

mehdiakiki · 2026-03-25T02:10:02Z

It seems a bit unfortunate that we're essentially implementing this macro twice (once as a decl macro, once as a Syn parser) but it's probably worth the benefit. Can we update the options! macro definition to mention that this parser needs to be kept up to date?

That's right. That's a concern to keep in mind. Should we consider adding a test that parses options! in order to catch potential changes to the macro?

clubby789 · 2026-03-27T12:39:40Z

I believe parses_real_unstable_options_file should catch that, this comment is just to hopefully catch issues before waiting for CI to run.
LGTM now, thanks
@bors r+

rust-bors · 2026-03-27T12:39:43Z

📌 Commit 75bd926 has been approved by clubby789

It is now in the queue for this repository.

Rollup of 10 pull requests Successful merges: - #154070 (Unstable book options parser) - #154371 (Use LocalDefId for more tcx method calls) - #154405 (Improve doc comment unicode guidance) - #154431 (Avoid ICE in explicit reference cast suggestion for unrelated leaf pr…) - #153528 (Fix LegacyKeyValueFormat report from docker build: mips) - #154246 (Add test for issue #101532: dead code warnings in const _) - #154421 (Rustdoc rejects html emits with json output) - #154428 (bootstrap: `-Zjson-target-spec` for synthetic targets) - #154437 (bootstrap.example.toml: Hint how to allow `build.warnings`) - #154454 (fix: [rustfmt] prevent panic when rewritng associated item delegations) Failed merges: - #154450 (Use the normal arg-parsing machinery for `-Zassert-incr-state`)

Rollup merge of #154070 - mehdiakiki:unstable-book-options-parser, r=clubby789 Unstable book options parser Parses the `options!` macro in `compiler/rustc_session/src/options.rs` directly to extract the unstable (-Z) compiler flag names and descriptions to generate documentation for the unstable book. I took notice from the previous attempt which ran `rustc -Zhelp` and parsed the output and used this approach that reads the source directly. Used claude for the tedious char by char parsing parts but verified the code, I hope that's ok!

Zalathar · 2026-03-28T02:28:49Z

I'm unhappy to see that this was merged, as it seems unreasonably burdensome to maintenance of unstable options.

Revert "Unstable book options parser" - Reverts rust-lang#154070 This reverts commit 6fd8466, reversing changes made to fda6d37. --- The changes in rust-lang#154070 are well-intentioned, but in their current form they place an unreasonable maintenance burden on the affected part of the compiler, out of proportion to the benefit gained. - Moving the unstable options to a separate `include!` file is marginally more convenient for the unstable-book-gen tool, but a big hassle for compiler contributors looking to read or modify the list of unstable options. It breaks normal file layout conventions, interferes with navigation, and interferes with IDE features, all without ever being truly necessary. - The extra `syn`-based parser in unstable-book-gen is quite complicated, almost entirely undocumented, and seemingly buggy. That makes it extremely difficult to understand or modify, which in turn makes it effectively impossible to ever change the `options!` macro on the compiler side. - The new tests in unstable-book-gen don't appear to be hooked up to run in CI. - The compiler changes don't appear to have been approved by a T-compiler reviewer. Before contemplating a revert, I tried to put together a fix-forward to resolve or mitigate some of these issues. But unstable-book-gen in its current state makes that impractical, so unfortunately I believe that the only way forward is to revert the original changes. --- I do think the reverted functionality is potentially useful, and I'm not fundamentally opposed to a revised version of it landing in the future, if the relevant concerns are addressed. But I feel strongly that the current version does not justify the real and troublesome maintenance burden that it imposes on compiler contributors.

Noratrieb · 2026-03-28T07:40:56Z

Wouldn't a much easier alternative be to just add a new perma-unstable --print flag that dumps machine readable information about the flags (like JSON)?

jyn514 · 2026-03-28T07:50:32Z

Then we have to build the compiler before building the unstable book. I guess that’s not the end of the world.

Rollup merge of #154488 - Zalathar:unstable, r=jieyouxu Revert "Unstable book options parser" - Reverts #154070 This reverts commit 6fd8466, reversing changes made to fda6d37. --- The changes in #154070 are well-intentioned, but in their current form they place an unreasonable maintenance burden on the affected part of the compiler, out of proportion to the benefit gained. - Moving the unstable options to a separate `include!` file is marginally more convenient for the unstable-book-gen tool, but a big hassle for compiler contributors looking to read or modify the list of unstable options. It breaks normal file layout conventions, interferes with navigation, and interferes with IDE features, all without ever being truly necessary. - The extra `syn`-based parser in unstable-book-gen is quite complicated, almost entirely undocumented, and seemingly buggy. That makes it extremely difficult to understand or modify, which in turn makes it effectively impossible to ever change the `options!` macro on the compiler side. - The new tests in unstable-book-gen don't appear to be hooked up to run in CI. - The compiler changes don't appear to have been approved by a T-compiler reviewer. Before contemplating a revert, I tried to put together a fix-forward to resolve or mitigate some of these issues. But unstable-book-gen in its current state makes that impractical, so unfortunately I believe that the only way forward is to revert the original changes. --- I do think the reverted functionality is potentially useful, and I'm not fundamentally opposed to a revised version of it landing in the future, if the relevant concerns are addressed. But I feel strongly that the current version does not justify the real and troublesome maintenance burden that it imposes on compiler contributors.

rustbot added A-attributes Area: Attributes (`#[…]`, `#![…]`) S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 19, 2026

rustbot assigned petrochenkov Mar 19, 2026

This comment has been minimized.

Sign in to view

mehdiakiki force-pushed the unstable-book-options-parser branch from 6d964c6 to 96b9265 Compare March 19, 2026 02:11

This comment has been minimized.

Sign in to view

mehdiakiki force-pushed the unstable-book-options-parser branch from 96b9265 to d5f2c91 Compare March 19, 2026 02:19

jyn514 reviewed Mar 19, 2026

View reviewed changes

rustbot added the T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. label Mar 19, 2026

rustbot assigned notriddle and unassigned petrochenkov Mar 19, 2026

rustbot added the T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) label Mar 19, 2026

rustbot assigned clubby789 and unassigned notriddle Mar 19, 2026

mehdiakiki added 4 commits March 21, 2026 12:02

docs: autogenerate compiler flag stubs by parsing options! macro

a22e9e0

fix: move tests to separate file and avoid tidy markers in strings

83a21a5

docs: parse unstable options with syn

b4f2c8a

style: fix rustfmt

deeb45a

mehdiakiki force-pushed the unstable-book-options-parser branch from b1a4466 to deeb45a Compare March 21, 2026 16:07

clubby789 reviewed Mar 23, 2026

View reviewed changes

address PR review feedback for unstable-book-gen

75bd926

rust-bors bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 27, 2026

This was referenced Mar 27, 2026

Rollup of 10 pull requests #154466

Merged

Use the normal arg-parsing machinery for -Zassert-incr-state #154450

Merged

rust-bors bot merged commit 6fd8466 into rust-lang:main Mar 28, 2026
11 checks passed

rustbot added this to the 1.96.0 milestone Mar 28, 2026

Zalathar mentioned this pull request Mar 28, 2026

Revert "Unstable book options parser" #154488

Merged

	"unexpected field count for option `{name}`: expected 4 or 5, found {}",
	"unexpected field count for option `{name}`: expected {REQUIRED_FIELDS} or {OPTIONAL_FIELDS}, found {}",

Uh oh!

Conversation

mehdiakiki commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Mar 19, 2026

Uh oh!

rustbot commented Mar 19, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

jyn514 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

jyn514 Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

mehdiakiki Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

mehdiakiki Mar 21, 2026

Choose a reason for hiding this comment

Uh oh!

petrochenkov commented Mar 19, 2026

Uh oh!

jyn514 commented Mar 19, 2026

Uh oh!

rustbot commented Mar 21, 2026

Uh oh!

clubby789 commented Mar 23, 2026

Uh oh!

clubby789 Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

mehdiakiki Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

rustbot commented Mar 25, 2026

Uh oh!

mehdiakiki commented Mar 25, 2026

Uh oh!

clubby789 commented Mar 27, 2026

Uh oh!

rust-bors bot commented Mar 27, 2026

Uh oh!

Uh oh!

Zalathar commented Mar 28, 2026

Uh oh!

Noratrieb commented Mar 28, 2026

Uh oh!

jyn514 commented Mar 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

mehdiakiki commented Mar 19, 2026 •

edited

Loading