-
Notifications
You must be signed in to change notification settings - Fork 12.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Don't make statement nonterminals match pattern nonterminals #120221
Don't make statement nonterminals match pattern nonterminals #120221
Conversation
r? @nnethercote (rustbot has picked a reviewer for you, use r? to override) |
compiler/rustc_ast/src/token.rs
Outdated
NtBlock(..) | | ||
NtPath(..)), | ||
/// **NB**: Take care when modifying this function, since it will change | ||
/// the stable set of tokens that are allowed to match an pat nonterminal. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/// the stable set of tokens that are allowed to match an pat nonterminal. | |
/// the stable set of tokens that are allowed to match a pat nonterminal. |
compiler/rustc_ast/src/token.rs
Outdated
AndAnd | // double reference | ||
Literal(_) | // literal | ||
DotDot | // range pattern (future compat) | ||
DotDotDot | // range pattern (future compat) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAICT, the net effect here:
- All identifiers now match, i.e. more than before
DotDotEq
no longer matchesBinOp(Or)
matches ifallow_leading_or
is trueNtExpr
,NtMeta
,NtTy
now matchNtBlock
no longer matches
Is that right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, though it should be noted that can_begin_pattern was previously only used for diagnostics and was previously just a rip off of can_begin_expr from above.
A better way to look at the changes here is to compare it to the previous match that was located in nonterminal_may_begin_with. This PR copies that logic over, adds an uninterpolate call for consistency with can_begin_expr (rather than manually handling the NtIdent arm), and adjusting the match arm for the Interpolated token.
To sum it up, I believe the only token kind that now no longer matches pat nonterminals is the statement nonterminal.
} | ||
} | ||
NonterminalKind::PatParam { .. } => token.can_begin_pattern(false), | ||
NonterminalKind::PatWithOr => token.can_begin_pattern(true), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And the net effect here is that NtStmt
no longer matches. Which I think is the change that actually fixes the original problem, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
How did you find this problem? Just curious. This appears to change more than is necessary to fix the cited bug. Is that intended? |
I noticed it while authoring #120218, and thinking a bit about why there was a
Yes. I don't see why we need to maintain an incorrect |
Thanks for the explanations. r=me with the "a"/"an" nit fixed. |
90ac96c
to
6917fc5
Compare
Actually, given that this is a change to a contract (the macro matchers), I'll pass this by t-lang as well. @rustbot label: +I-lang-nominated |
☔ The latest upstream changes (presumably #122830) made this pull request unmergeable. Please resolve the merge conflicts. |
compiler/rustc_ast/src/token.rs
Outdated
NtPath(..)), | ||
/// **NB**: Take care when modifying this function, since it will change | ||
/// the stable set of tokens that are allowed to match a pat nonterminal. | ||
pub fn can_begin_pattern(&self, allow_leading_or: bool) -> bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the bool here is very non-obvious in callers. Dunno if it'd be worth having an enum or something to make it clearer, or to have the bool-taking thing be non-pub with different names for the two cases, or something.
From CE: Right now the tokens that a macro matcher may begin with is a stable guarantee. We are relaxing the assumption that pattern matchers may begin with statement metavariables ($var whose type is stmt), because when we actually try to parse such a pattern, we are always guaranteed to fail. This only allows more code to compile, and would only break future code if we specifically wanted to begin patterns with statement metavariable. I agree that it's weird to allow a @rfcbot fcp merge The other thing we explored was what it would take to make this actually work, since you can actually put an |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
From CE: Right now the tokens that a macro matcher may begin with is a stable guarantee. We are relaxing the assumption that pattern matchers may begin with statement metavariables ($var whose type is stmt), because when we actually try to parse such a pattern, we are always guaranteed to fail. This only allows more code to compile, and would only break future code if we specifically wanted to begin patterns with statement metavariable. I agree that it's weird to allow a @rfcbot fcp merge The other thing we explored was what it would take to make this actually work, since you can actually put an |
Team member @scottmcm has proposed to merge this. The next step is review by the rest of the tagged team members: No concerns currently listed. Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up! cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns. |
@bors r+ |
…-patterns, r=nnethercote Don't make statement nonterminals match pattern nonterminals Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to `may_be_ident`: https://github.com/rust-lang/rust/blob/ef71f1047e04438181d7cb925a833e2ada6ab390/compiler/rustc_parse/src/parser/nonterminal.rs#L21-L37 This has the unfortunate side effect that a `stmt` nonterminal eagerly matches against a `pat` nonterminal, leading to a parse error: ```rust macro_rules! m { ($pat:pat) => {}; ($stmt:stmt) => {}; } macro_rules! m2 { ($stmt:stmt) => { m! { $stmt } }; } m2! { let x = 1 } ``` This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal. As a side-effect, I modified `Token::can_begin_pattern` to work correctly and used that in `Parser::nonterminal_may_begin_with`.
…kingjubilee Rollup of 9 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#127912 (std: make `thread::current` available in all `thread_local!` destructors) - rust-lang#128166 (Improved `checked_isqrt` and `isqrt` methods) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129366 (linker: Synchronize native library search in rustc and linker) - rust-lang#129527 (Don't use `TyKind` in a lint) - rust-lang#129534 (Deny `wasm_c_abi` lint to nudge the last 25%) - rust-lang#129640 (Re-enable android tests/benches in alloc/core) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) r? `@ghost` `@rustbot` modify labels: rollup
…-patterns, r=nnethercote Don't make statement nonterminals match pattern nonterminals Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to `may_be_ident`: https://github.com/rust-lang/rust/blob/ef71f1047e04438181d7cb925a833e2ada6ab390/compiler/rustc_parse/src/parser/nonterminal.rs#L21-L37 This has the unfortunate side effect that a `stmt` nonterminal eagerly matches against a `pat` nonterminal, leading to a parse error: ```rust macro_rules! m { ($pat:pat) => {}; ($stmt:stmt) => {}; } macro_rules! m2 { ($stmt:stmt) => { m! { $stmt } }; } m2! { let x = 1 } ``` This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal. As a side-effect, I modified `Token::can_begin_pattern` to work correctly and used that in `Parser::nonterminal_may_begin_with`.
…kingjubilee Rollup of 8 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129366 (linker: Synchronize native library search in rustc and linker) - rust-lang#129527 (Don't use `TyKind` in a lint) - rust-lang#129534 (Deny `wasm_c_abi` lint to nudge the last 25%) - rust-lang#129640 (Re-enable android tests/benches in alloc/core) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129731 (Allow running `./x.py test compiler`) r? `@ghost` `@rustbot` modify labels: rollup
…kingjubilee Rollup of 8 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129366 (linker: Synchronize native library search in rustc and linker) - rust-lang#129527 (Don't use `TyKind` in a lint) - rust-lang#129534 (Deny `wasm_c_abi` lint to nudge the last 25%) - rust-lang#129640 (Re-enable android tests/benches in alloc/core) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129731 (Allow running `./x.py test compiler`) r? `@ghost` `@rustbot` modify labels: rollup
…kingjubilee Rollup of 8 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129366 (linker: Synchronize native library search in rustc and linker) - rust-lang#129527 (Don't use `TyKind` in a lint) - rust-lang#129534 (Deny `wasm_c_abi` lint to nudge the last 25%) - rust-lang#129640 (Re-enable android tests/benches in alloc/core) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129731 (Allow running `./x.py test compiler`) r? `@ghost` `@rustbot` modify labels: rollup
…kingjubilee Rollup of 8 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129366 (linker: Synchronize native library search in rustc and linker) - rust-lang#129527 (Don't use `TyKind` in a lint) - rust-lang#129534 (Deny `wasm_c_abi` lint to nudge the last 25%) - rust-lang#129640 (Re-enable android tests/benches in alloc/core) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129731 (Allow running `./x.py test compiler`) r? `@ghost` `@rustbot` modify labels: rollup
…-patterns, r=nnethercote Don't make statement nonterminals match pattern nonterminals Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to `may_be_ident`: https://github.com/rust-lang/rust/blob/ef71f1047e04438181d7cb925a833e2ada6ab390/compiler/rustc_parse/src/parser/nonterminal.rs#L21-L37 This has the unfortunate side effect that a `stmt` nonterminal eagerly matches against a `pat` nonterminal, leading to a parse error: ```rust macro_rules! m { ($pat:pat) => {}; ($stmt:stmt) => {}; } macro_rules! m2 { ($stmt:stmt) => { m! { $stmt } }; } m2! { let x = 1 } ``` This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal. As a side-effect, I modified `Token::can_begin_pattern` to work correctly and used that in `Parser::nonterminal_may_begin_with`.
…kingjubilee Rollup of 10 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#127897 (add `aarch64_unknown_nto_qnx700` target - QNX 7.0 support for aarch64le) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129642 (Bump backtrace to 0.3.74~ish) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129723 (Simplify some extern providers) - rust-lang#129724 (Remove `Option<!>` return types.) - rust-lang#129725 (Stop using `ty::GenericPredicates` for non-predicates_of queries) - rust-lang#129733 (Subtree update of `rust-analyzer`) - rust-lang#129754 (wasi: Fix sleeping for `Duration::MAX`) r? `@ghost` `@rustbot` modify labels: rollup
…-patterns, r=nnethercote Don't make statement nonterminals match pattern nonterminals Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to `may_be_ident`: https://github.com/rust-lang/rust/blob/ef71f1047e04438181d7cb925a833e2ada6ab390/compiler/rustc_parse/src/parser/nonterminal.rs#L21-L37 This has the unfortunate side effect that a `stmt` nonterminal eagerly matches against a `pat` nonterminal, leading to a parse error: ```rust macro_rules! m { ($pat:pat) => {}; ($stmt:stmt) => {}; } macro_rules! m2 { ($stmt:stmt) => { m! { $stmt } }; } m2! { let x = 1 } ``` This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal. As a side-effect, I modified `Token::can_begin_pattern` to work correctly and used that in `Parser::nonterminal_may_begin_with`.
…kingjubilee Rollup of 9 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129642 (Bump backtrace to 0.3.74~ish) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129723 (Simplify some extern providers) - rust-lang#129724 (Remove `Option<!>` return types.) - rust-lang#129725 (Stop using `ty::GenericPredicates` for non-predicates_of queries) - rust-lang#129733 (Subtree update of `rust-analyzer`) - rust-lang#129751 (interpret/visitor: make memory order iteration slightly more efficient) r? `@ghost` `@rustbot` modify labels: rollup
…kingjubilee Rollup of 9 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129642 (Bump backtrace to 0.3.74~ish) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129723 (Simplify some extern providers) - rust-lang#129724 (Remove `Option<!>` return types.) - rust-lang#129725 (Stop using `ty::GenericPredicates` for non-predicates_of queries) - rust-lang#129733 (Subtree update of `rust-analyzer`) - rust-lang#129751 (interpret/visitor: make memory order iteration slightly more efficient) r? `@ghost` `@rustbot` modify labels: rollup
…kingjubilee Rollup of 9 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129642 (Bump backtrace to 0.3.74~ish) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129723 (Simplify some extern providers) - rust-lang#129724 (Remove `Option<!>` return types.) - rust-lang#129725 (Stop using `ty::GenericPredicates` for non-predicates_of queries) - rust-lang#129733 (Subtree update of `rust-analyzer`) - rust-lang#129751 (interpret/visitor: make memory order iteration slightly more efficient) r? `@ghost` `@rustbot` modify labels: rollup
…-patterns, r=nnethercote Don't make statement nonterminals match pattern nonterminals Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to `may_be_ident`: https://github.com/rust-lang/rust/blob/ef71f1047e04438181d7cb925a833e2ada6ab390/compiler/rustc_parse/src/parser/nonterminal.rs#L21-L37 This has the unfortunate side effect that a `stmt` nonterminal eagerly matches against a `pat` nonterminal, leading to a parse error: ```rust macro_rules! m { ($pat:pat) => {}; ($stmt:stmt) => {}; } macro_rules! m2 { ($stmt:stmt) => { m! { $stmt } }; } m2! { let x = 1 } ``` This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal. As a side-effect, I modified `Token::can_begin_pattern` to work correctly and used that in `Parser::nonterminal_may_begin_with`.
…iaskrgr Rollup of 15 pull requests Successful merges: - rust-lang#120221 (Don't make statement nonterminals match pattern nonterminals) - rust-lang#126183 (Separate core search logic with search ui) - rust-lang#129123 (rustdoc-json: Add test for `Self` type) - rust-lang#129366 (linker: Synchronize native library search in rustc and linker) - rust-lang#129527 (Don't use `TyKind` in a lint) - rust-lang#129534 (Deny `wasm_c_abi` lint to nudge the last 25%) - rust-lang#129640 (Re-enable android tests/benches in alloc/core) - rust-lang#129642 (Bump backtrace to 0.3.74~ish) - rust-lang#129675 (allow BufReader::peek to be called on unsized types) - rust-lang#129723 (Simplify some extern providers) - rust-lang#129724 (Remove `Option<!>` return types.) - rust-lang#129725 (Stop using `ty::GenericPredicates` for non-predicates_of queries) - rust-lang#129731 (Allow running `./x.py test compiler`) - rust-lang#129751 (interpret/visitor: make memory order iteration slightly more efficient) - rust-lang#129754 (wasi: Fix sleeping for `Duration::MAX`) r? `@ghost` `@rustbot` modify labels: rollup
Rollup merge of rust-lang#120221 - compiler-errors:statements-are-not-patterns, r=nnethercote Don't make statement nonterminals match pattern nonterminals Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to `may_be_ident`: https://github.com/rust-lang/rust/blob/ef71f1047e04438181d7cb925a833e2ada6ab390/compiler/rustc_parse/src/parser/nonterminal.rs#L21-L37 This has the unfortunate side effect that a `stmt` nonterminal eagerly matches against a `pat` nonterminal, leading to a parse error: ```rust macro_rules! m { ($pat:pat) => {}; ($stmt:stmt) => {}; } macro_rules! m2 { ($stmt:stmt) => { m! { $stmt } }; } m2! { let x = 1 } ``` This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal. As a side-effect, I modified `Token::can_begin_pattern` to work correctly and used that in `Parser::nonterminal_may_begin_with`.
Right now, the heuristic we use to check if a token may begin a pattern nonterminal falls back to
may_be_ident
:rust/compiler/rustc_parse/src/parser/nonterminal.rs
Lines 21 to 37 in ef71f10
This has the unfortunate side effect that a
stmt
nonterminal eagerly matches against apat
nonterminal, leading to a parse error:This PR fixes it by more accurately reflecting the set of nonterminals that may begin a pattern nonterminal.
As a side-effect, I modified
Token::can_begin_pattern
to work correctly and used that inParser::nonterminal_may_begin_with
.