Consider newlines as spaces for space-sensitive parsing

[TotalVerb]

Fix #16594 by treating newlines, tabs, etc. as spaces that make binary parsing take precedence over unary parsing in space-sensitive situations.

cc @stevengj

[stevengj]

Looks good to me, but @JeffBezanson should probably take a look, too.

[stevengj]

Note that #16594 occurred only for the special case of operators that are parsed as both unary and binary, so a more conservative version of this PR would just change the unary-and-binary-ops case.

[TotalVerb]

This issue manifested in a different way for colon. It's minor, but the new behaviour is better.

Before:

julia> [1 :
       2]
1×2 Array{Int64,2}:
 1  2

Now:

julia> [1 :
       2]
ERROR: syntax: line break in ":" expression

(this case is tested)

As for ~, it is both a unary and binary operator, just a strange one, so #16594 does indeed apply.

Old

julia> :(@test 1 ~
               2)
ERROR: syntax: missing comma or ) in argument list

New

julia> :(@test 1 ~
               2)
:(@test @~(1,2))

[TotalVerb]

The initial version of this PR was incorrect, leading to test failures. I have corrected this PR.

[StefanKarpinski]

Failure is macOS-only. Probably unrelated.

[StefanKarpinski]

bump @JeffBezanson

[tkelman]

Jeff, please review, do we want this one or not?

[stevengj]

Bump.

[JeffBezanson]

This example from the linked issue seems like the biggest problem:

julia> [1 + 
       2 0]
1×2 Array{Int64,2}:
 3  0

(where the trailing space on the first line makes a difference). I wonder if it's possible to fix just that?
@stevengj Do you want to take a look at this? Otherwise I feel we're running out of time and we can't really deal with this.

[stevengj]

I fixed the merge conflict.

[stevengj]

The change seems pretty reasonable to me. Breaking, but unlikely to affect working real-world code. It is 11th hour but this one has been hanging around for a decision for a while…

[StefanKarpinski]

Failure is real—does not pass parsing tests.

[TotalVerb]

I think the issue is that the introduced space-before-next-token? consumes the whitespace unlike the existing peek-char test. It can probably be resolved by testing the next character against the spaces without skipping them? I forget why I introduced the skip-ws now. I can take a quick look.

[TotalVerb]

Hold on, it looks like all the test failures are precisely the added tests (which are testing the issue that this fixes) which have bitrotted over time. The argument count for each of them is precisely off by one because the macro syntax has changed. The string one must have had a few extra quotes once upon a time. Let's see if the new commit fixes it.

[stevengj]

The :([x .+ y]) parsing had changed to a vect node, so I fixed that test (I hope) as well.

[TotalVerb]

One last test it looks like... This one because of ParseError binding deprecation.

[TotalVerb]

@StefanKarpinski good to go now?

[StefanKarpinski]

macOS failure is #26725. This isn't my call—@JeffBezanson will have to take a look and decide.

[JeffBezanson]

I would prefer these tests to check more than just the number of arguments.

[stevengj]

Yeah, just check Expr equality?

[TotalVerb]

OK, I have improved the tests using @stevengj's suggestion.