NFC normalisation of lifetimes and labels

[mattheww]

The lexer NFC-normalises identifiers, as described in rfc2457. For example, K (LATIN CAPITAL LETTER K) and K (KELVIN SIGN) are treated as representing the same identifier.

But this doesn't happen for lifetimes or labels, so 'K and 'K are different as lifetimes or labels.

For example, this compiles (without warning) in Rust 1.79, while this doesn't compile.

I'm not sure whether this should be considered a bug.

rfc2457 doesn't explicitly mention lifetimes or labels.

The Reference describes the current behaviour: it treats identifiers and lifetimes/labels as parallel forms of token, and documents normalisation as happening for identifiers only.

On the other hand it seems surprising, and some of the discussion around #126452 has expressed the view that "lifetimes are identifiers".

[petrochenkov]

This should certainly be fixed and identifiers in lifetimes should be normalized in the same way as identifiers in other places.

In proc macro API, for example, there are no lifetime tokens and lifetimes are two tokens - a quote and an identifier (with Joint on the first one), so lifetimes coming from proc macros are indeed normalized.
Proc macro API is public interface to this part of the language, observable by users, so rustc's lexer gluing 'ident into a single token can be treated as an internal detail, which shouldn't change behavior such as Unicode normalization

The normalization of lifetimes was likely just missed when the normalization was originally implemented, and nobody noticed it until now, because it's a pretty exotic detail of the language.
@mattheww Could you actually make a PR fixing this in rustc?

[gmorenz]

@rustbot claim