RFC: Extend the |> operator in order to chain tuples.

[Ismael-VC]

I was trying to use an annonymous function in place, something like this:

julia> (2, 3, 4) |> (x, y, z) -> 2x + 3y^2 * 4x/2
ERROR: wrong number of arguments
 in anonymous at none:1

This works but is less readable:

julia> ((x, y, z) -> 2x + 3y^2 * 4x/2)(2, 3, 4)
112.0

Extending |> to map tuples of arguments:

julia> Base.|>(xs::Tuple, f) = f(xs...)
|> (generic function with 7 methods)

This will let us make arbitrary chaining possible:

julia> (2, 3, 4) |> (x, y, z) -> 2x + 3y^2 * 4x/2
112.0

Previous usage and semantics are left unchanged as far as I can tell, also see the tests and documentation.

[tkelman]

no it isn't? docstrings need to be put in .jl code for this to be accurate

[Ismael-VC]

I'm sorry, I was away for a while, and haven't catch up to recent doc changes. I just copied it over. Where do I need to put this .jl code then, for this to be true?

[Ismael-VC]

Oooh I get it now! :P

[tkelman]

It's explained in CONTRIBUTING.md

[tkelman]

I disagree with your readability comparison between the standard function application syntax used by the language vs the chaining pun of the |> operator, which I'd personally rather deprecate as not idiomatic. This addition might be harmless though, not sure. see below

[vtjnash]

see #13240

[Ismael-VC]

@tkelman I admit that my examples are short of imagination, here are good examples of idiomatic julian code in the wild, in which the logic is best read from left to right:

• At Escher:

Moreover, using the curried version with the |> infix operator makes for code that reads better.

function main(window)
    tex("T = 2\\pi\\sqrt{L\\over g}") |>
        fontcolor("#499") |>
        pad(5mm) |>
        fillcolor("#eeb")
end

You can mentally read this as: to the plaintext Hello, World, apply font color red and then apply a padding of 10mm.

|> as it is now in operators.jl is crippled and useless for me, so I instinctively extend it, and share.

We have seen that people expect this to work. The stack overflow example in #13240 is one that I have been asked about a lot.

When asked What's in your .juliarc.jl file? Stefan answers:

Mine's empty. I just take anything I want in .juliarc.jl and put it directly in base. j/k

I think I'm with him on this one, why have every one extend |> on their own?

This addition might be harmless though, not sure

That's what I also think, not sure either. When I first tried this I used ..., the other PR solves this.

But mine conveys the idea that |> maps a tuple of arguments to a function as is and seems a little bit more general, I'll be doing more tests and examples.

[Ismael-VC]

At the same What's in your .juliarc? thread:

Haha, I actually thought about something related when posting the question: there's a lot of common boilerplate in people's .vimrc files, because Vim is stuck carrying around a lot of backwards-compatibility baggage. But if something is enough of a pain point to be in most everyone's .juliarc.jl, at this stage of development it'll probably get added in base soon enough (unless it's stuff like using SOME_PACKAGE_I_ALWAYS_USE like in the linked message).

• SundaraRaman R

[vtjnash]

This addition might be harmless though, not sure.

x |> f would now mean either f(x) or f(x...) depending on whether x is a Tuple

[tkelman]

That looks like Escher is pretending Julia is much more of a functional language than it actually is. It doesn't make a lot of sense to me to be returning functions from things like fontcolor, etc, if that's what is happening there. This argument has happened in other function chaining issues, I think it only really makes sense for single-input single-output functions, and doesn't combine well with multiple dispatch. It's a natural idiom in languages that don't have multiple dispatch or are designed with passing functions around as a well optimized use case, but in Julia it looks and feels wrong IMO.

[jballanc]

Instead of invoking currying or returning functions from fontcolor, etc., chaining with |> could be achieved by re-writing the method calls, similar to Clojure's threading operators (->, cond->, and friends). My problem with this is that using a macro for chaining feels less natural in Julia than in Clojure due to lack of S-exps (e.g. @-> foo(x), bar(y), baz(z)). It would be better if we could have infix macros (e.g. foo(x) @-> bar(y) @-> baz(z)), but that's a whole other subject.

[tkelman]

Infix macros was #11608, which was closed with a "this is probably not a good idea, but you could try to implement it if you really want to." Macros don't play well with extending to new types since they work only on expressions, so packages would clobber each other's definitions if they both tried to define the same infix macro with different meanings.

[Ismael-VC]

@tkelman

That looks like Escher is pretending Julia is much more of a functional language than it actually is.

I don't think there are any pretensions, Escher's code works, and several people like this style, which they use and extend since we encourage the use of |>, that is a fact.

I'd like to hear @shashi opinion on this PR.

I disagree with your readability comparison between the standard function application syntax used by the language vs the chaining pun of the |> operator

Why do you think it's a pun and/or dislike this syntax?, I'm truly curious. I agree that the |> operator is a pun as it is defined right now in Base (|>(x, f) = f(x) and that's it!).

I think it only really makes sense for single-input single-output functions, and doesn't combine well with multiple dispatch.

I don't understand why you say so, ie:

julia> Base.|>(xs::Tuple, f) = f(xs...)
|> (generic function with 7 methods)

julia> foo(x) = "calling foo(x)"
foo (generic function with 1 method)

julia> foo(x, y) = "calling foo(x, y)"
foo (generic function with 2 methods)

julia> foo(x::Int, y::Char) = "calling foo(x::Int, y::Char)"
foo (generic function with 3 methods)

julia> foo{T<:Number}(x::T, y::T) = "calling foo{T<:Number}(x::T, y::T)"
foo (generic function with 4 methods)

julia> foo(x, y, args...) = "calling foo(x, y, args...)"
foo (generic function with 5 methods)

julia> 1 |> foo
"calling foo(x)"

julia> (1,) |> foo
"calling foo(x)"

julia> (1, "2") |> foo
"calling foo(x, y)"

julia> (1, 2) |> foo
"calling foo{T<:Number}(x::T, y::T)"

julia> (1, '2') |> foo
"calling foo(x::Int, y::Char)"

julia> (1, '2', "3", [4:6;]) |> foo
"calling foo(x, y, args...)"

[Ismael-VC]

@vtjnash

x |> f would now mean either f(x) or f(x...) depending on whether x is a Tuple

Yes, what I meant when I said:

Previous usage and semantics are left unchanged as far as I can tell, also see the tests and documentation.

I meant that the previous usage of f(x) would remain unaffected (no one needs to rewrite code because of this), of course, this would make the behaviour of |> dependant in wether x is a Tuple or not. I think that's good, that's why we have multiple dispatch don't we?

It would be weird if x could be an Vector and then |>(x::Vector, f) meant f(x...). But:

Tuples are an abstraction of the arguments of a function — without the function itself.

So it makes sense to me for |> to map and apply a function to the preceding argument or tuple of arguments. This would actually allow for easy (n-args/multiple dispatch) function chaining.

[Ismael-VC]

I know currying has been previously discussed, with no consensus on sight, but this PR is not about a new fancy way of implementing currying either using macros or what else, it's just about extending the current |> operator (with one LOC), in order avoid every one else extending it on their own right now (with potential bugs).

Here is another evidence of people expecting this to work, it's in spanish so I didn't bring it to the table at first:

• https://gitter.im/JuliaLangEs/julialang-es?at=55dd34ce004c3e375ad0c212

Currently support for currying has been:

1. First: |>(x, f::Function) = f(x)
2. Then: |>(x, f::Callable) = f(x)
3. Finally: |>(x, f) = f(x)

And that's it! But it is there. So I agree that if we won't really support this (why?) it should be removed from Base, but in contrast to my PR, people would actually need to rewrite their code and so it's a bigger breaking change.

[tkelman]

Escher's code works, and several people like this style

Has anyone benchmarked against code to do the same task but written idiomatically rather than currying everything? I'd never write numerical code like that.

I dislike the syntax because it's backwards and I'd rather open source code be written with a mostly consistent style given that more people will be using and reading it than writing it. Having two syntaxes for calling functions that each look backwards and wrong to different subsets of the community is bad, I'd rather pick one style and stick with it. Pipelines work for bash but don't really work for Julia. Given the inference issues with function arguments, the performance penalty of splatting, and the semantic issue @vtjnash points out, I'm against merging this change.

[Ismael-VC]

Has anyone benchmarked against code to do the same task but written idiomatically rather than currying everything? I'd never write numerical code like that.

@tkelman here are some tests, there is of course overhead, but I don't think is that much, I understand that you wouldn't write numerical code like this (I wouldn't either), but not all code is numerical code, Escher is an example. I expect it to be slow if it's using anonymous functions (this is going to change AFAIK), but that hasn't stop them to use it, they care more about the style it is read (think DSL, isn't Julia supposed to be great at those?):

julia> @time 1 |> foo;
  0.000004 seconds (4 allocations: 160 bytes)

julia> @time foo(1);
  0.000002 seconds (4 allocations: 160 bytes)

julia>

julia> @time (1,) |> foo;
  0.000007 seconds (5 allocations: 176 bytes)

julia> @time foo((1,)...);
  0.000007 seconds (5 allocations: 176 bytes)

julia>

julia> @time (1, "2") |> foo;
  0.000005 seconds (5 allocations: 192 bytes)

julia> @time foo((1, "2")...);
  0.000005 seconds (5 allocations: 192 bytes)

julia> @time foo(1, "2");
  0.000002 seconds (4 allocations: 160 bytes)

julia>

julia> @time (1, 2) |> foo;
  0.000007 seconds (5 allocations: 192 bytes)

julia> @time foo((1, 2)...);
  0.000006 seconds (5 allocations: 192 bytes)

julia> @time foo(1, 2) ;
  0.000002 seconds (4 allocations: 160 bytes)

julia>

julia> @time (1, '2') |> foo;
  0.000006 seconds (5 allocations: 192 bytes)

julia> @time foo((1, '2')...);
  0.000007 seconds (6 allocations: 208 bytes)

julia> @time foo(1, '2');
  0.000003 seconds (4 allocations: 160 bytes)

julia>

julia> @time (1, '2', "3", [4:6;]) |> foo;
  0.000014 seconds (7 allocations: 336 bytes)

julia> @time foo((1, '2', "3", [4:6;])...);
  0.000012 seconds (8 allocations: 352 bytes)

julia> @time foo(1, '2', "3", [4:6;]);
  0.000005 seconds (6 allocations: 288 bytes)

I agree with you about consistency, there are a lot of:

foo() = begin
    # ...lots of code
end

In Julia's source code, which should be IMHO:

function foo()
    # ...lots of code
end

But even if we enforce that style, I don't think that banning the first option would be nice nor a good idea (it's great at the REPL, where one do more writing than reading).

Perhaps a Currying (using Currying)package would be a nice place to explore this syntax, without compromising Julia's semantics.

Just to be sure, are you against this PR only or also against retaining |>. I can do another PR to remove it, but I wonder about other's opinions.

Type inference is the worst issue:

julia> @code_warntype 1 |> foo;
Variables:
  x::Int64
  f::F

Body:
  begin  # operators.jl, line 198:
      return (f::F)(x::Int64)::Any
  end::Any

julia> @code_warntype foo(1);
Variables:
  x::Int64

Body:
  begin  # none, line 1:
      return "calling foo(x)"
  end::ASCIIString

Do you think this is not going to change along with the splatting slowness (I've been told not to use it ...a lot lately!)?

[Ismael-VC]

By the way, Could anyone tell me why is F the type of f (f::F)? Thanks!

[tkelman]

I don't think we'd have consensus on totally deprecating |> at this time, but it could be discouraged as not idiomatic if we have an unofficial style guide somewhere. #13412 may change some of the performance of this idiom, but it won't change the aesthetics which are more subjective.

If you can find and fix any foo() = begin I think that would be a good cleanup.

A real DSL would be implemented using macros, where it's free to do whatever it wants with syntax as long as it parses. DSL's that cause parse errors can be implemented as string macros, like Cxx.jl.

The splatting penalty probably won't go away completely, since it forces the call to go through the Julia multiple dispatch machinery. Maybe if the splatted collection's length is known at compile time then more optimizations could be done.

[jballanc]

@tkelman Honestly just curious... if |> is considered non-idiomatic, what is idiomatic? foo(bar(baz(qux(), 2), 'a'), :hello) does not, in my opinion, seem any clearer than qux() |> baz(2) |> bar('a') |> foo(:hello). Is there another approach to nested application like this that I'm missing? Assigning each intermediate result to a new variable seems like overkill.

[tkelman]

The former is the syntax for function calls, as evidenced by the pun implementation of |>. The latter is ambiguous, would you put the placeholder in the first argument or last? Or some intermediate argument? See #5571 for a bunch of ideas on this, some of which have been implemented out in packages with macros. Pipe.jl, Lazy.jl, FunctionalDataUtils.jl, etc. It's a matter of opinion here, but one style is always going to read as ugly to a non-negligible group of users. The question is which is least ugly overall and the least punny in implementation.

[Ismael-VC]

I'd never write numerical code like that.

@tkelman could you tell me if you would recommend the use splat at all in numerical code? I wonder because recently I've noticed that everyone promptly advises me not to use it, no mater how banal the example I use is. Are you OK with using it in non numerical code?

[tkelman]

In that quote I was referring to the currying API example you gave from Escher, not splatting specifically. Splatting is convenient and there are plenty of situations where it's the nicest-looking way to accomplish something, but it comes with a performance penalty that should be kept in mind is all. It's not really suitable for anything that's going to be performance-critical. If you don't care at all about performance then you can use whatever code style you want, but performance is probably the largest single selling point with Julia.

[shashi]

here are good examples of idiomatic julian code in the wild, in which the logic is best read from left to right:

@Ismael-VC I would not call this idiomatic Julia at all, it's something specific to Escher's DSL. The curried methods are generated by a macro (which has other odd features like optional leading arguments). There is absolutely no need to complicate the base language with such features.

@tkelman, yes, fontcolor returns a lambda which applies the color to its argument.

This argument has happened in other function chaining issues, I think it only really makes sense for single-input single-output functions, and doesn't combine well with multiple dispatch.

This is very true, in Escher I am cautious to avoid such situations while using the said macro. It is not and should not be the concern of the user of the library.

That looks like Escher is pretending Julia is much more of a functional language than it actually is.

Rather, I'd like to think of Julia as a small, flexible substrate for doing anything.

@Ismael-VC I don't think this PR should go in, because it adds a finicky special case to an otherwise very clear |> operator like @vtjnash pointed out. Instead tackling something like #6614 could alleviate some of your concerns that motivated this PR.

[Ismael-VC]

Thank you everyone for your comments, very educative as always!

[Ismael-VC]

@tkelman The type inference issues are not introduced by this PR, that's the way it is now, so I don't see why adding the tuple method is something bad, but the current method isn't (it's used in Julia base) taking into account only the inference of course, ie. not the semantic meanings.

I wonder about this, because don't you think this could be another good target for a cleanup?, ie, change all occurrences in julia source of:

foo |> bar to bar(foo)

Since it's more performant and inference is more accurate, If someone think it's a good idea I'd like to take care of that clean up too.

[tkelman]

I actually have a branch where I did just that - 212727c, it's a few months out of date and needs a rebase, I think I got distracted by other things before opening a PR for it.

[Ismael-VC]

That's great! I'll take care of the other cleanup then. Maybe we can reach consensus to deprecate |> then, once it's usage is removed from base.

[Ismael-VC]

Here is yet another user trying to pass multiple arguments to |>:

I'm a bit puzzled by the behavior of the pipe operator when feeding values to functions expecting multiple arguments. Basically it doesn't seem to work at all. Am I missing something?

• At julia-users: Pipe operator and functions with multiple arguments

[hayd]

So... should |> be deprecated and moved to a package where it can be extended?

Also: I suspect @one-more-minute will have an opinion on this / |>.

[MikeInnes]

@hayd thanks for the ping. I do disagree with some of the arguments against this PR; for example, I don't think it's a good idea to be dogmatic about coding style. Fundamentally, low-level systems code should be written differently from high-level logic like web routers, and I personally think it's a great strength that Julia supports these different levels of abstraction reasonably fluidly.

(In general, if I find code difficult to read it's because the concepts behind it are difficult, not because I'm confused by x |> f or surprising whitespace settings – so worrying about those things strikes me as a premature optimisation.)

That said, this change means that the semantics of the |> function change depending on the types of its arguments. This is something we've learned to avoid in Julia, because semantics should be predictable from syntax alone. #13240 seems like a better way to achieve this (although I also wonder how often it will be useful in practice).

[Ismael-VC]

@hayd If we are not going to extend it in Base, then I think it's a good idea to deprecate and move it to a package, it's causing to many _WTF_s as it is IMHO. And I really like the syntax and want to use it sometimes instead of deep nested calls or having to think about temporary variables. Package developers would be able to extend, use and most importantly: share this code and make it really useful and general, implementing it as they see fit.

But if the type inference problems can be fixed, I think |> is worth having in Base. I don't agree with the argument that |> behavior is very clear:

it adds a finicky special case to an otherwise very clear |> operator like @vtjnash pointed out

It has been proven that it's not very clear to everyone, maybe changing the docstring:

Applies a function to the preceding argument. This allows for easy function chaining.

to something that clearly states:

Applies a function to the preceding argument. This allows for single-inputs/single-ouputs functions chaining.

Or at least remove the easy part. After reading all your opinions, I also disagree with this being or not being idiomatic. Who gets to decide that? Could we at least make a poll or something more in the spirit of the Quorum language (The world's first evidence-oriented programming language)?

Bringing randomized controlled trials to human factors decisions in programming language design. You know ... science.

We could really leverage to crowd source opinions.

Also even if the meaning of the method |>(x, f) of the generic function |> where actually very clear. I still don't see what's wrong with adding another method to it (even more to a lang in development, where not everything is set into stone)? This is very odd to hear for me since we have generic functions with dozens of methods, and the manual states:

The ability to define function behavior across many combinations of argument types via multiple dispatch

The only show stopper for me so far is the type inference issue (we have survived with it in this case, so far), but I understand this is going to be fixed.

[Ismael-VC]

@one-more-minute I actually wanted to do what #13240 solves, but it only works at the beginning of the expression AFAICT:

julia> Base.|>(x, y, f) = f(x, y)
|> (generic function with 7 methods)

julia> Base.|>(args...) = args[end](args[1:end-1]...)
|> (generic function with 8 methods)

julia> (1, 2)... |> complex |> x -> (x.re, x.im)
(1,2)

julia> (1, 2)... |> complex |> (x -> (x.re, x.im))... |> divrem    # people might try this
ERROR: MethodError: `start` has no method matching start(::Function)
 in append_any at essentials.jl:127

julia> (1, 2)... |> complex |> (x -> (x.re, x.im))()... |> divrem    # or this
ERROR: UndefVarError: x not defined

julia> Base.|>(xs::Tuple, f) = f(xs...)
|> (generic function with 9 methods)

julia> (1, 2) |> complex |> x -> (x.re, x.im) |> divrem
(0,1)

semantics should be predictable from syntax alone

I understand but my solution seems to be more generic.

[hayd]

If we are not going to extend it in Base, then I think it's a good idea to deprecate and move it to a package

+1

A related issue IMO is function composition, it'd be nice if there was an efficient multi-argument solution for that too. #4963 (comment)

[JeffBezanson]

I agree with @vtjnash here. You can't switch to splatting based on the type of the argument. f(x) is supposed to be "generic" in the parametric sense that it doesn't matter what x is.

[JeffBezanson]

Bringing randomized controlled trials to human factors decisions in programming language design.
You know ... science.

I think this is a problematic idea. I definitely see the value for relatively superficial decisions like how to spell keywords, and that you should use infix syntax for math instead of lisp syntax, etc. However I think the results on this are fairly unsurprising, and programming languages have been converging on them already.

But these decisions are only a tiny part of language design. A human subject sitting for a brief session on language syntax will not be sensitive to many important factors. Perhaps the most obvious one is what happens when a project gets big and continues for a long time.

The Stefik and Siebert paper is quite careful and explicit about narrowing the scope to, essentially, keyword choices. It's a fine result but it hardly gives you a full language design. One has to appreciate just how unimportant many syntax choices are. For example, Quorum seems to use : instead of . for method calls. I'm willing to believe novices prefer this, but how much more productive does it really make them? Is the gain so big that it's worth overturning a convention? Does the productivity gain hold when you also need to learn and use existing languages that use .?

I notice Quorum uses the syntax Array<integer> for type parameters. Needless to say I'm skeptical of whether human psychology really demands using the "less than" and "greater than" symbols to delimit type parameters.

Quorum is a java-like, class-based, imperative language where the classes have "actions". Does that mean "the science says" people shouldn't learn or use functional programming? No, it means the authors are asking "what language does the evidence lead us to, assuming it must look a lot like java?"

[Ismael-VC]

@JeffBezanson how can anyone say something is julian or not if its taking into account only personal opinions or the opinions of a relatively small group of people, when there are many other julians out there with their own (unknown) opinions that make up the community and are unaware of this discussions but may be willing to share their opinions?

...assuming it must look a lot like java?"

How is that different from?

...assuming it must look a lot like matlab?"

I understand that you disagree with their design choices and that is perfectly fine, but I think you miss the point. I'm talking about the evidence, the culture and spirit of quorum, science, a small group of people taking better and more informed decisions that benefit the whole julian community in an easy way, not in the current implementation of the Quorum language.

[JeffBezanson]

How is that different from? ...assuming it must look a lot like matlab?

The main difference is that I don't claim that superficial features of julia are evidence-based. But the only point of my comment is that human-factors-based evidence only gets you so far.

The problem is this:

record_result(x) = print(file, x)

...

run(x, y, z) |> record_result

Clearly, the intent is to record whatever result we get. However, if one time the result happens to be a tuple, you get a no method error. This is surprising, especially because you can currently always rewrite record_result(run(x, y, z)) to run(x, y, z) |> record_result. There is an appealing simplicity to something that always works. Remembering exceptions is burdensome.

You claimed that nobody would need to rewrite code because of this. That is false. Uses of x |> f where x might be a tuple are broken by this change. Am I missing something?

[hayd]

@Ismael-VC Sometimes you'll still need parens...

julia> ((1, 2)... |> complex |> (x -> (x.re, x.im)))... |> divrem
(0,1)

We can't/don't need to change the semantics for passing tuples.

---

I think the consensus is that |> is not idiomatic julia (but it's great the language supports doing this kind of thing), whether that means it should be deprecated I'm not sure - but let's discuss it in a seperate issue/PR.

Discussion of changing how language decisions are made is way off-topic.

[Ismael-VC]

@hayd oh of course, thank you for pointing that out, it would be difficult to keep up the parenthesis in a multiline expression though.

Uses of x |> f where x might be a tuple are broken by this change. Am I missing something?

@JeffBezanson yes I see what you mean now, I was the one how missed that.

julia> foo(x::Tuple) = @show x
julia> Base.|>(xs::Tuple, f) = f(xs...)
|> (generic function with 7 methods)

julia> x = (1, 2, 3)
(1,2,3)

julia> foo(x::Tuple) = @show x
foo (generic function with 1 method)

julia> foo(x)
x = (1,2,3)
(1,2,3)

julia> x |> foo
ERROR: MethodError: `foo` has no method matching foo(::Int64, ::Int64, ::Int64)
 in |> at none:1

julia> (x,) |> foo
x = (1,2,3)
(1,2,3)

I can make another PR in order to deprecate |> is that ok everyone?

[mschauer]

Can we now consider #13240 again which is non-breaking and solves the problems with multiple arguments if there is no intent to abolish |>?