parse more dot operators, parse .= with assignment precedence #17393

stevengj · 2016-07-13T02:24:42Z

For 0.6, the plan in #16285 is to have a .+ b etcetera automatically correspond to broadcast(+, a, b), and be fused with other "dot calls". Also, we are planning to have x .= ... etcetera turn into broadcast! for in-place fused loops.

As a first step towards that goal, this PR:

Changes the precedence of .= to be assignment-like, rather than comparison-like.
Makes (nearly) every operator "dottable". The exceptions are operators with very special syntactic meanings, like $ and ||. (Some of these could be allowed, but I wanted to be conservative to start with).

I'd ideally like to get this merged in 0.5. The reason is that, once 0.6 rolls around, we will want a @compat macro to make some of #16285's vectorized syntax work (albeit suboptimally) in 0.5, and it will be hard to do that unless the operators parse.

stevengj · 2016-07-13T02:26:34Z

(I don't think we should document the new operators, yet, because we don't really want people to start assigning methods to them in 0.5 ... since in 0.6 the broadcast methods will be defined automatically given a method for the non-dotted operator.)

stevengj · 2016-07-13T02:27:49Z

As far as I can tell, no existing packages use .=, so the change in precedence should be fairly safe. In 0.6, .= will be a (broadcasting) assignment operator so it will not be possible to add methods to it anyway.

StefanKarpinski · 2016-07-13T02:30:26Z

👍

JeffBezanson · 2016-07-13T02:37:19Z

It seems like we could allow .& and .|.

stevengj · 2016-07-13T12:16:58Z

Rebased and added .& and .|

stevengj · 2016-07-13T12:24:23Z

@JeffBezanson, I'm bugged by .!. As I understand it, in order to parse .!= as an operator, the parser requires the prefix .! to also be parsed as an operator, which is why .! has to be an operator.

However, the .! operator was parsed with prec-comparison, i.e. as a binary operator with comparison precedence. I kept this behavior, but it seems weird. Shouldn't we parse .! as a unary operator if we parse it at all? Or would that screw up parsing of .!=?

JeffBezanson · 2016-07-13T15:01:15Z

We could handle that by rejecting .! at the end of read-operator. That is already done for --.

stevengj · 2016-07-13T15:11:20Z

Or maybe we should just allow unary operators to be dotted, and allow .! as a unary op? So that .!x would turn (in 0.6) into broadcast((!), x)?

This seems more consistent, but maybe it requires special handling in the parser to accept a unary .! and a binary .!=?

stevengj · 2016-07-13T15:17:28Z

(It seems like -- should nowadays be a prec-plus operator, similar to ++.)

stevengj · 2016-07-13T18:58:25Z

Okay, I took Jeff's suggestion and reject .! in the parser, like --. Just adding dots before unary operators didn't work (seems to require some deeper changes in the parser for make to succeed), so I punted on that for now. Seems better to err on the side of conservatism when adding new operators, anyway.

With unary operators the need for a dot prefix is somewhat less, because you can always write (!).(array), but it would still be nicer to support .!array eventually.

stevengj · 2016-07-14T06:52:14Z

Seems like the libgit2 Travis test failure must be unrelated?

stevengj · 2016-07-14T19:18:07Z

Restarting checks since the Travis failure on libgit2 should hopefully have been fixed by 74cf1b1, thanks @tkelman

stevengj · 2016-07-14T19:24:44Z

Rebased and added NEWS.

tkelman · 2016-07-14T20:13:35Z

Should this have any parsing tests?

andyferris · 2016-07-15T00:38:51Z

I think having .-, .!, etc as unary operators would make a lot of sense, but I understand you said it might be hard to implement.

I can currently do -v and (-).(v), so it seems odd I that can't do .-v.

stevengj · 2016-07-15T07:38:30Z

@tkelman, added some tests.

stevengj · 2016-07-15T20:20:24Z

@andyferris, I don't think it should be that hard to implement more dotted unary operators, it is just that we are so close to the 0.5 release that I don't want to dig deep into the parser right now.

stevengj · 2016-07-15T20:20:29Z

Okay to merge?

tkelman · 2016-07-16T05:10:52Z

lgtm but I'm the wrong person to review the scheme code

stevengj · 2016-07-16T11:08:28Z

@JeffBezanson, looks like it's up to you to hit the "merge" button?

StefanKarpinski · 2016-07-18T20:17:46Z

@JeffBezanson is on vacation, so I'd recommend merging if you're happy with it and we can do a post-merge review when he gets back.

vtjnash · 2016-07-18T21:43:30Z

lgtm. do these deparse (print) correctly?

(also, there's now a trivial merge conflict in test/core, so needs a rebase)

vtjnash · 2016-07-18T21:44:03Z

test/core.jl

@@ -4462,3 +4462,11 @@ end
 f17147(::Tuple) = 1
 f17147{N}(::Vararg{Tuple,N}) = 2
 @test f17147((), ()) == 2
+
+# PR #17393


put this in test/parse?

stevengj · 2016-07-19T17:39:28Z

@vtjnash, yes, the new dotted binary operators print fine. The assignment syntax x .= y prints as an Expr object, but since x .+= y in Julia 0.4 also prints as an Expr object I think that is fine (not a regression anyway)... we should fix printing of dotted assignments once this actually does something.

mbauman · 2016-07-20T23:09:17Z

Oooh, this is cool. Any thoughts here on decoupling the precedence of .| and .& from | and & to support a use case like #5187? That'd be something that'd be good to do in the same cycle that they're introduced since precedences are very difficult to change.

stevengj · 2016-07-20T23:30:06Z

@mbauman, I find #5187 very hard to follow because of the length of the thread. I would suggest making a PR that changes the precedence, and summarize in the PR what you think are the main arguments in favor from #5187.

tkelman · 2016-07-21T00:06:00Z

one note out of this is that we could deprecate elementwise | now that .| is a general version.

KristofferC · 2016-07-21T00:19:46Z

Did this PR break the following? I haven't bisected but it seems like the .| could be affected by this PR.

julia> Base.|>(a, b) = a+b
ERROR: syntax: ">(a,b)" is not a valid function argument name
 in eval(::Module, ::Any) at ./boot.jl:234
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46

(found from Gtk.jl stopped parsing):

stevengj · 2016-07-21T00:59:04Z

The dot operators are now parsed, but the don't do anything different from 0.4 in this PR.

Deprecations will have to wait until 0.6 when dot operators are sugar for a broadcast.

KristofferC · 2016-07-21T01:09:41Z

Ok, I guess something else made that syntax fail to parse because it worked in 0.4 and it is currently failing. That fact that the error message says that >(a,b) is not a valid function name (the .| part being removed from the error message) made me suspect that this PR had something to do with it,

stevengj · 2016-07-21T01:44:11Z

@KristofferC, sorry, I was replying to @tkelman. Probably this PR broke Base.|>. You should now write Base.:|>, the same way that you have to write Base.:+ and not Base.+

parsing of these was changed on JuliaLang/julia#17393 ref http://pkg.julialang.org/detail/IntervalConstraintProgramming.html

the parsing here was changed in JuliaLang/julia#17393

tkelman · 2016-07-21T02:42:08Z

Also broke IntervalConstraintProgramming and LibExpat

…#17393)" This reverts commit 814c974.

parsing of these was changed on JuliaLang/julia#17393 ref http://pkg.julialang.org/detail/IntervalConstraintProgramming.html

…ang#17393) * parse more dot operators for JuliaLang#16285, parse .= with assignment precedence * include .|= and .&= in syntactic-operators list * reject .! operator * NEWS for new operators * tests * move tests to parse.jl

…Base.≤

stevengj force-pushed the more-dotops branch from 9aba392 to 93b7fd9 Compare July 13, 2016 12:16

stevengj closed this Jul 14, 2016

stevengj reopened this Jul 14, 2016

stevengj force-pushed the more-dotops branch from b001d98 to a50927d Compare July 14, 2016 19:23

stevengj mentioned this pull request Jul 14, 2016

Vectorization Roadmap #16285

Closed

5 tasks

tkelman added the parser Language parsing and surface syntax label Jul 16, 2016

vtjnash reviewed Jul 18, 2016
View reviewed changes

stevengj force-pushed the more-dotops branch from 335ccb4 to fb1f962 Compare July 18, 2016 23:30

move tests to parse.jl

2450de8

stevengj force-pushed the more-dotops branch from 8b49130 to 2450de8 Compare July 19, 2016 15:21

stevengj merged commit 814c974 into JuliaLang:master Jul 19, 2016

stevengj deleted the more-dotops branch July 19, 2016 18:28

stevengj mentioned this pull request Jul 20, 2016

treat .= as syntactic sugar for broadcast! #17510

Merged

4 tasks

KristofferC mentioned this pull request Jul 21, 2016

fix broken syntax on 0.5 JuliaGraphics/Gtk.jl#245

Merged

tkelman mentioned this pull request Jul 21, 2016

import operators from Base instead of using Base. syntax JuliaIntervals/IntervalConstraintProgramming.jl#22

Merged

tkelman added a commit to JuliaIO/LibExpat.jl that referenced this pull request Jul 21, 2016

import | from Base instead of extending it via Base.|

0e44d8c

the parsing here was changed in JuliaLang/julia#17393

tkelman mentioned this pull request Jul 21, 2016

import | from Base instead of extending it via Base.| JuliaIO/LibExpat.jl#47

Merged

stevengj added a commit that referenced this pull request Jul 21, 2016

note that #17393 now requires one to use Base.:≤ instead of Base.≤

5bd48d1

KristofferC added a commit that referenced this pull request Jul 23, 2016

Revert "parse more dot operators, parse .= with assignment precedence (…

989dcab

…#17393)" This reverts commit 814c974.

KristofferC mentioned this pull request Aug 2, 2016

syntax: invalid operator ".!" JuliaLang/JuliaParser.jl#60

Closed

stevengj added the broadcast Applying a function over a collection label Aug 2, 2016

This was referenced Aug 2, 2016

& does not broadcast, and there is no .& (similar problem for "or") #6915

Closed

Allow users to define "dot" vectorized operators. #14544

Closed

mfasi pushed a commit to mfasi/julia that referenced this pull request Sep 5, 2016

note that JuliaLang#17393 now requires one to use Base.:≤ instead of …

12ce6b6

…Base.≤

ararslan mentioned this pull request Jan 17, 2017

Add .& and .| JuliaLang/Compat.jl#306

Merged

stevengj mentioned this pull request Feb 9, 2017

parsing ambiguity of .! ? #20541

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parse more dot operators, parse .= with assignment precedence #17393

parse more dot operators, parse .= with assignment precedence #17393

stevengj commented Jul 13, 2016

stevengj commented Jul 13, 2016

stevengj commented Jul 13, 2016

StefanKarpinski commented Jul 13, 2016

JeffBezanson commented Jul 13, 2016

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 13, 2016 •

edited

Loading

JeffBezanson commented Jul 13, 2016

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 13, 2016

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 14, 2016

stevengj commented Jul 14, 2016 •

edited

Loading

stevengj commented Jul 14, 2016

tkelman commented Jul 14, 2016

andyferris commented Jul 15, 2016

stevengj commented Jul 15, 2016

stevengj commented Jul 15, 2016 •

edited

Loading

stevengj commented Jul 15, 2016

tkelman commented Jul 16, 2016

stevengj commented Jul 16, 2016

StefanKarpinski commented Jul 18, 2016

vtjnash commented Jul 18, 2016

vtjnash Jul 18, 2016

stevengj Jul 18, 2016

stevengj commented Jul 19, 2016 •

edited

Loading

mbauman commented Jul 20, 2016

stevengj commented Jul 20, 2016

tkelman commented Jul 21, 2016

KristofferC commented Jul 21, 2016 •

edited

Loading

stevengj commented Jul 21, 2016 •

edited

Loading

KristofferC commented Jul 21, 2016

stevengj commented Jul 21, 2016 •

edited

Loading

tkelman commented Jul 21, 2016

parse more dot operators, parse .= with assignment precedence #17393

parse more dot operators, parse .= with assignment precedence #17393

Conversation

stevengj commented Jul 13, 2016

stevengj commented Jul 13, 2016

stevengj commented Jul 13, 2016

StefanKarpinski commented Jul 13, 2016

JeffBezanson commented Jul 13, 2016

stevengj commented Jul 13, 2016 • edited Loading

stevengj commented Jul 13, 2016 • edited Loading

JeffBezanson commented Jul 13, 2016

stevengj commented Jul 13, 2016 • edited Loading

stevengj commented Jul 13, 2016

stevengj commented Jul 13, 2016 • edited Loading

stevengj commented Jul 14, 2016

stevengj commented Jul 14, 2016 • edited Loading

stevengj commented Jul 14, 2016

tkelman commented Jul 14, 2016

andyferris commented Jul 15, 2016

stevengj commented Jul 15, 2016

stevengj commented Jul 15, 2016 • edited Loading

stevengj commented Jul 15, 2016

tkelman commented Jul 16, 2016

stevengj commented Jul 16, 2016

StefanKarpinski commented Jul 18, 2016

vtjnash commented Jul 18, 2016

vtjnash Jul 18, 2016

Choose a reason for hiding this comment

stevengj Jul 18, 2016

Choose a reason for hiding this comment

stevengj commented Jul 19, 2016 • edited Loading

mbauman commented Jul 20, 2016

stevengj commented Jul 20, 2016

tkelman commented Jul 21, 2016

KristofferC commented Jul 21, 2016 • edited Loading

stevengj commented Jul 21, 2016 • edited Loading

KristofferC commented Jul 21, 2016

stevengj commented Jul 21, 2016 • edited Loading

tkelman commented Jul 21, 2016

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 13, 2016 •

edited

Loading

stevengj commented Jul 14, 2016 •

edited

Loading

stevengj commented Jul 15, 2016 •

edited

Loading

stevengj commented Jul 19, 2016 •

edited

Loading

KristofferC commented Jul 21, 2016 •

edited

Loading

stevengj commented Jul 21, 2016 •

edited

Loading

stevengj commented Jul 21, 2016 •

edited

Loading