Lexical analysis of identifiers

[mattheww]

The main part of the syntax rules for identifiers in §2.3 (fls_21vnag69kbwe) is as follows:

PureIdentifier ::=
    XID_Start XID_Continue*
  | _ XID_Continue+

NonKeywordIdentifier ::=
    PureIdentifier
  | WeakKeyword

RawIdentifier ::=
    r# (PureIdentifier | RawIdentifierKeyword)

Identifier ::=
    NonKeywordIdentifier
  | RawIdentifier

Defining "pure identifier"

This seems to have been garbled somewhere.

NonKeywordIdentifier is defined as PureIdentifier | WeakKeyword, but PureIdentifier doesn't do anything to exclude keywords, so PureIdentifier and NonKeywordIdentifier have identical meaning and include all keywords.

The definition of "pure identifier" in the text is different, and excludes weak keywords but not strict or reserved keywords.

§2.3:3 (fls_xsdmun5uqy4c)

An identifier is a lexical element that refers to a name.

§2.3:4 (fls_ktnf6zkrdy45)

A pure identifier is an identifier that does not include weak keywords.

The upshot is that the FLS says Identifier includes at least strict and reserved keywords, and so Name does too, which is certainly wrong.

I think the intent must have been that both "pure identifier" and PureIdentifer would exclude all keywords.

Lone underscore

In the FLS's model identifier is evidently intended to exclude lone underscore (which is treated as punctuation).

The syntax rules do say that, but the description of the Unicode profile in §2.3:5 (fls_jpecw46eh061) allows lone underscore.

The fix for the parallel problem in the Reference was rust-lang/reference#1129.

Zero width characters

§2.3:9 (fls_cs6cbw625np1) reads

Characters 0x200C (zero width non-joiner) and 0x200D (zero width joiner) shall not appear in a pure identifier.

This paragraph isn't saying anything new: those characters aren't included in XID_Start or XID_Continue so they wouldn't be allowed anyway.

There is a similar paragraph in the Reference.