parser bug: macro calls w/o parens in generator expressions

[StefanKarpinski]

julia> maximum([@elapsed sleep(1) for k = 1:3])
1.003554001

julia> maximum(@elapsed(sleep(1)) for k = 1:3)
1.006058621

julia> maximum(@elapsed sleep(1) for k = 1:3)
ERROR: syntax: unexpected ")"

[c42f]

Hmm. We want

:(@a k for k=1:2)            -> :(@a(k) for k=1:2)              (1)
:(@a k for k=1:2; k ; end)   -> :(@a(k, for k=1:2; k ; end))    (2)

Currently the comprehension version works by setting the inside-vect parser state to true so it can stop parsing the macro when seeing a 'for token in parse-space-separated-exprs. But this simple strategy won't work for generators because case (2) above is already valid syntax.

[JeffBezanson]

Yes, I'm not sure this can be resolved. We use syntax like @threads x for i = 1:n ... where the for loop is an argument to the macro. The only possibility is to parse it differently inside parentheses, but then we have an expression that can't be parenthesized without changing its parsing, which seems like a bad idea to me.

[c42f]

Yes I imagined an inside-parens state might make sense and something inside parse-space-separated-exprs would just need to look further ahead to determine whether a given occurrence of for is a prefix of a for loop, or a generator expression.

But I'm not sure that kind of look ahead is compatible with the structure of the parser? It rather looks like it isn't.

[JeffBezanson]

Looked into this a bit. I think there are basically three options:

1. Do nothing --- only being inside square brackets can stop a macro from grabbing a for expression.
2. Sacrifice (a; b; c) block syntax, as described in parser bug: macro calls w/o parens in generator expressions #18650 (comment). (@mac for i in x ... ) would always parse as a generator. Real-world example of code that would break:

broadcast!(f, X::AbstractArray) = (@inbounds for I in eachindex(X); X[I] = f(); end; X)

We could make @mac for (when the for loop is the first argument to the macro) special, but I think that's too fiddly.

3. Change the parsing only inside call syntax, so f(@mac for i = ...) would pass a generator to f.

I think (3) is a pretty good option, as it fixes the example in the OP, and there is no real use for writing a for loop in a function argument list. However, the difference between function arguments and tuple arguments might be confusing.

[StefanKarpinski]

I guess 3 does sound like the best option, followed by 1. I would be sad to lose (a; b; c) syntax.

[JeffBezanson]

Potential change in #22943