Specification and Grammar incomplete with regards to lexing rules

[bruno-medeiros]

Both the reference (http://doc.rust-lang.org/reference.html#lexical-structure) and the grammar reference (http://doc.rust-lang.org/grammar.html#lexical-structure) are very incomplete with regards to the lexing rules. For example:

• lifetimes token not documented
• attributes token not documented
• character/string literal token rules are incomplete, they do not seem to specify the b/br# prefixes for character and string literals.

I'm writing a lexer for an IDE (to do syntax highlighting), and since I'm not yet familar with Rust itself, this makes my life a bit harder...

[steveklabnik]

Yup, we'd love to get this filled out.

[fhahn]

@bruno-medeiros there exists an Antler lexer, which only differs from the Rust lexer slightly, maybe this could help.

[bruno-medeiros]

there exists an Antler lexer, which only differs from the Rust lexer slightly, maybe this could help.

Yeah it does. Another user had also directed me to that.

Hum, and I also just realized the Attribute rule is not a token, but a grammar rule...

[steveklabnik]

I'm going to close this in favor of #30942, which is more general.