[css-syntax-3] Tokenize comments into whitespaces

[cdoublev]

The current version of Firefox accepts calc(1/**/+ 1) or calc(1 +/**/1) whereas the current version of Chrome does not. Per spec, Chrome is correct.

There is no test for this on WPT. Based on discussions in the www-style public archives, there was an intention to relax the whitespace requirement in the future.

Both browsers accept :nth-child(+/**/n) as a selector but not U/**/+0-1 as a unicode-range. The related specs do not accept a whitespace at the place of the comment.

Has it ever been considered to tokenize comments into whitespaces?

This would make these cases more consistent. The non-normative requirement for the serialization of "any consecutive pair of tokens" (not separated by whitespaces) in CSS Syntax 3 § 9. Serialization might then no longer be necessary.

I suspect the breaking change for <an+b> is ok. The unicode range tests on WPT still expect comments to be valid within the range, but I presume it will be updated at some point (related: #8835).

Also, has it ever been considered to consume a run of whitespaces/comments into a single <whitespace-token>?

There is currently no guarantee that a token stream does not include consecutive whitespaces. For example, the result from tokenizing 1 /**/ 1 includes two consecutive whitespaces.

This would not fix any problem but since consecutive whitespaces are collapsed, this seems consistent.