AST-72: correctly lex verbatim unicode

[ggreif]

See issue: https://dfinity.atlassian.net/browse/AST-72

The main problem was that the unicode function fetched a byte from the string, interpreted it as a code point and encoded it as a singleton string into UTF-8.

Now we only do this when it is an ASCII char, otherwise we pick up all non-ASCII chars into a substring, convert that to code-points and pick the first one from the sequence, converting it back to UTF-8 encoded string.

[ggreif]

I'll try to find the Wasm.Utf8 module's interface.

[nomeata]

This looks very much through the chest into the eye. You don't have to use Wasm.Utf8 if it does not provide what you need, and parsing utf8 is actually simple: The first byte tells you how many bytes make up the code point (if it is not ASCII, then count leading 1 to know the number of bytes), so you know far to read (up to four bytes). You can just code up each of these cases, doing the correct shifts and or to assemble the code point. See this table.

[crusso]

Hmm, I wonder if the real problem here is that we are using ocamllex, which isn't Unicode friendly, rather than ulex or some other Unicode aware lexer. However, if the solution in the PR works, I'm happy to go for it.

[ggreif]

This looks very much through the chest into the eye.

@nomeata Yes, now I analyse the leader UTF-8 byte. See commit 9e653c9.

[nomeata]

I left some ideas for further polishing, if you feel inspired, but nothing that I'd require before merging.

[nomeata]

I guess it is a bit codesmelly to use a function that returns a string to parse a single character, but it's fine.

Maybe strengthen the pattern match to [code]? It should return exactly one code after all, right?

[ggreif]

I guess it is a bit codesmelly to use a function that returns a string to parse a single character, but it's fine.

If you only have a hammer (Utf8.decode) then a substring looks like the perfect nail :-)

[ggreif]

Maybe strengthen the pattern match to [code]? It should return exactly one code after all, right?

Yes, this is a good idea, now that I only look at one Unicode code point.