[Clarity] Escaped backslash in strings may fail parsing

[njordhov]

Describe the bug

Parsing a string with escaped backslash \\ sometimes fails with error.

Steps To Reproduce

Evaluate in the Clarity repl:
(list "\\" "x")
=> error: Expected whitespace or a close parens. Found: 'x'

A contract with the code above also fails to deploy on testnet.

Expected behavior

Strings with an escaped backslash should parse correctly.

Note that the expression parses if the "x" is removed. Evaluate in the Clarity repl:

(list "\\")
=> ["\\"]

Environment (please complete the following information):

Confirmed in the latest Clarity REPL and by deploying to Testnet.

[obycode]

I can dig into this one. Upon first inspection, it looks like the v2 parser in use by the REPL handles it, but the v1 parser (the canonical parser) does not, which would mean it would need to be fixed along with other consensus breaking changes in 2.1. I will take a closer look and verify that later today.

[obycode]

The problem is in this regular expression from vm/ast/parser/mod.rs:140:

        LexMatcher::new(
            r##""(?P<value>((\\")|([[ -~]&&[^"]]))*)""##,
            TokenType::StringASCIILiteral,
        ),

For the ascii string in the example, (list "\\" "x"), the first token in the list from the lexer is the string "\\\" ".

We have the improved parser v2 in the REPL which handles this correctly. 2.1 seems like a good time to switch the canonical VM over to use this new parser. Should I begin putting together a PR for that?

[jcnelson]

Closing as complete if this was fixed in the new v2 parser. Please reopen if not.

[njordhov]

@jcnelson Is there a unit test to validate it has been fixed for the v2 parser?

[obycode]

No, I don't think there is. You're right, we should add tests for these.

[njordhov]

There should also be tests for related cases that parses but shouldn't, such as:

(list "\\" x")

[blockstack-devops]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.