Lexical syntax simplification

[emberian]

Rendered

[emberian]

Another benefit of this is that the output of the lexer can be only spans and their associated token type, rather than having to do any work.

[lilyball]

This can't just be 'r' LIT_STR_RAW_INNER2? (and the inner tokens should probably be swapped).

[emberian]

Indeed it can.

[emberian]

This needs to take into account rust-lang/rust#14400 still.

[chris-morgan]

The exponent [0-9]* part—should it be [0-9_]*?

Also I think this will be tightening what is accepted; at present, for example, 1. is acceptable (but not 1.f32 for clear reasons), but this change will break that. Is that deliberate? Desirable? &c.

[emberian]

Good catch. That was not deliberate, I didn't mean to change the float literal syntax at all.

[pcwalton]

+1, sounds like an improvement for rustfmt

[emberian]

@kballard is the CRLF stuff correct? I extended the places that accept newline to also accept '\r\n', but not '\r', and I've removed '\r' from the whitespace skipping.

[lilyball]

@cmr My patch actually allows bare '\r' in whitespace skipping and in non-doc comments. It only rejects it inside of strings and doc comments. That said, I don't know if it's worth trying to be that permissive. It may be better just to go ahead and treat a bare '\r' without a subsequent '\n' as a hard error anywhere in the file.

[lilyball]

Shouldn't [^'\n\t\r] be ~['\n\t\r]?

[lilyball]

Also the character set needs to include \\ or else invalid character escapes will end up matching anyway.

[zwarich]

This grammar is wildly ambiguous. Identifiers, numbers and operators can be tokenized in multiple ways.

[huonw]

@zwarich do you have an example of an ambiguous sequence of tokens?

[emberian]

"12.12" could be INTEGER(12) DOT INTEGER(12)

[emberian]

etc. will be relatively easy to fix.

[lilyball]

Doesn't antlr4 pick the longest matching token?

[huonw]

(Yeah, isn't the maximal munch principal the standard way to resolve "ambiguities" like this?)

[zwarich]

@kballard @huonw Yes, that is the standard way of resolving ambiguities with lexical syntax, and apparently the 'lexer grammar' feature of ANTLR makes it choose this strategy, as opposed to what it uses for normal grammars.

[anasazi]

I like the idea of keeping comments after lexing so pretty-printers / refactoring tools can use the same lexer as the compiler, but how about we just make comment dropping a micropass between the lexer and parser instead of adding to the parser workload?

[emberian]

Sure, whatever.

[emberian]

Fixed most things, and verified that it works as I expect.

[emberian]

cc @nikomatsakis @pcwalton @brson I've updated this. It behaves as I expect for the code I've run it against, and accepts/rejects everything it should in the compiler/libs/testsuite/servo.

[brson]

Accepted as RFC 21 per https://github.com/mozilla/rust/wiki/Meeting-weekly-2014-05-27#lexer-changes