Reserve guarded string literals (RFC 3593)

[pitaj]

Implementation for RFC 3593, including:

• lexer / parser changes
• diagnostics
• migration lint
• tests

We reserve #", ##", ###", ####, and any other string of four or more repeated #. This avoids infinite lookahead in the lexer, though we still use infinite lookahead in the parser to provide better forward compatibility diagnostics.

This PR does not implement any special lexing of the string internals:

• strings preceded by one or more # are denied
• regardless of the number of trailing #
• string contents are lexed as if it was just a bare "string"

Tracking issue: #123735
RFC: rust-lang/rfcs#3593

[rustbot]

r? @estebank

rustbot has assigned @estebank.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[mattheww]

In edition 2024 I'm seeing a panic inside cook_unicode if I give this an empty guarded string without enough closing hashes (eg #"" or ##""#).

[mattheww]

In edition 2024 if I give this a string like blah"xx" I get both a warning for an unknown prefix and a warning for an unprefixed guarded string literal.

I think it would be better if I got only the first warning.

[pitaj]

@mattheww good catch with the empty strings. I wasn't able to reproduce the "double-warning" issue with blah"xx", but just managed to do so with blah#"xx"#.

I think it's actually the correct behavior. When you fix the issue with blah #"xx"# as recommended, you'll end up getting the second error anyways. Better to get both at once, then you can make a more educated decision.

Regardless, that case will be exceedingly rare if it exists in the wild at all, so I'm not going to spend any time on it.

[mattheww]

My stress-tester is now giving an ICE for £#""# in editions older than 2024.

[mattheww]

My lexer stress-tests don't find any problems with this now.

(They're checking for ICEs, cases where the lexer output changes in previous editions, and cases where the lexer output is not as expected in the 2024 edition.)

[pitaj]

The crater run for #124605 found some real regressions. So that proves the effort made here for backwards compatibility is warranted.

[traviscross]

We reviewed this in the edition call today. What's the next step here, e.g. is this waiting on further review or on updates from @pitaj?

cc @estebank

[estebank]

@traviscross I'm just concerned with introducing the snapshotting behavior into the lexer. There's no precedent for that approach in that stage of the compiler. It might be that there's no alternative, but would like more people in @rust-lang/compiler to double check on the implementation.

[oli-obk]

cc @Nilstrieb @compiler-errors fun lexer things

[Noratrieb]

forwarding my ping to @nnethercote

[pitaj]

@rustbot ready

[traviscross]

@bors r+

[bors]

📌 Commit 321a5db has been approved by traviscross

It is now in the queue for this repository.

[bors]

⌛ Testing commit 321a5db with merge f708670...

[rust-log-analyzer]

The job x86_64-gnu-llvm-18 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

##[group]Building stage1 library artifacts (x86_64-unknown-linux-gnu)
##[endgroup]
##[group]Building tool rustdoc (stage0 -> stage1, x86_64-unknown-linux-gnu)
##[endgroup]
thread 'core::builder::tests::ci_rustc_if_unchanged_logic' panicked at src/core/builder/tests.rs:260:5:
assertion failed: has_changes == config.download_rustc_commit.is_none()
   0: rust_begin_unwind
             at /rustc/8c27a2ba6b21f3406a51118643080f0591949827/library/std/src/panicking.rs:662:5
   1: core::panicking::panic_fmt
             at /rustc/8c27a2ba6b21f3406a51118643080f0591949827/library/core/src/panicking.rs:74:14

[bors]

💔 Test failed - checks-actions

[pitaj]

That failure looks unrelated. Can we try rerunning bors?

[ehuss]

@bors retry

CI error will be fixed by #131461.

[bors]

⌛ Testing commit 321a5db with merge df1b5d3...