Remove require, import, and maybe importall and merge into using

[quinnj]

From the mailing list:

Stefan:

I think we should do away with import altogether and just have

using Foo
using Foo: bar

The first would load and create a binding for Foo, make it's exports available as "soft" bindings (what using does now). The second would also load and create a binding for Foo, and make bar available as a "hard" binding (what import does now)."

Seems this is still quite a point of confusion for newcomers.

[IainNZ]

We'd need the import Foo functionality too somehow, where you just get the Foo and nothing else

[simonster]

using Foo:?

[vtjnash]

using Foo: Foo?

[ssfrr]

To bind Foo to the Foo module (and nothing else):
import Foo
To bind Foo to module Foo, x to Foo.x, y to Foo.y
import Foo: x, y
To bind Foo to module Foo, and all Foo's exported names are bound unqualified:
import Foo: *

This could be using instead, but I feel like this is more in the spirit of import.

This also removes the distinction between bringing something into scope and making it available for method extension. I personally think that's a good thing and makes the module system more understandable, but I wanted to make sure we brought it up.

[simonster]

There is a strong case to be made that a construct that makes all of a module's exported bindings available should make them soft (using) bindings and not hard (importall) bindings. Suppose module A uses module B and defines foo(::Any). If a later release of module B also defines and exports foo(::Any), you don't want them to clobber each other. You also don't want module B to define foo(::Int) and have module A sometimes call that method instead of the method it defined because it is more specific, or to have to list all of the identifiers you want from module B to avoid importing a single conflicting identifier.

But if you have explicitly listed the identifiers you want to import, then there is never a reason to give a soft binding. Either you are not going to define new methods of an identifier, in which case hard and soft bindings have identical behavior, or you are going to define new methods of it, in which case a soft binding is equivalent to no binding and if you want that behavior you should just remove the identifier from the list.

So, in short, I like @StefanKarpinski's proposal. Conceptually we need both hard and soft bindings, but we can get all useful behaviors with a single keyword.

[ssfrr]

I see your point. In that case I like your proposal that using Foo: (with the colon but no items) seems conceptually consistent. The colon indicates that you're going to limit which symbols to pull in, but you just don't list any.

The empty trailing : looks a little funny, but I think people would get used to it

[StefanKarpinski]

The trouble with the empty trailing colon is that we currently look for a name on the next line – in other words using Foo: is taken to be incomplete.

[quinnj]

Related: #4600

[toivoh]

I know that current Julia practice is to import everything into the
namespace of the current module, but I really think that there should still
be a simple and natural way to import just the name of a module.

I also think that the current overloading of using Foo and using Foo.Bar is
problematic; it looks inside Foo but not inside Foo.Bar (unless Foo.Bar is
a module?)

[kmsquire]

I think in Stefan's proposal, this is addressed with

using Foo: Foo

On Thursday, August 14, 2014, toivoh [email protected] wrote:

I know that current Julia practice is to import everything into the
namespace of the current module, but I really think that there should still
be a simple and natural way to import just the name of a module.

I also think that the current overloading of using Foo and using Foo.Bar is
problematic; it looks inside Foo but not inside Foo.Bar (unless Foo.Bar is
a module?)

—
Reply to this email directly or view it on GitHub
#8000 (comment).

[staticfloat]

@kmsquire but what then happens if there is a Foo module inside of the Foo module? That's unfortunately ambiguous.

[vtjnash]

That's a design error (and causes a warning), so it doesn't matter

[BobPortmann]

I like the version proposed by @ssfrr above the best.

[trans]

Is there an underlying convention that packages have one and only one module entry point? -- To import Foo means import module Foo of package Foo. Any other module must be a submodule of Foo. Which means there is an implicit correspondence between package Foo, Foo.jl in the package and the module Foo within it.

I ask b/c I am wondering if import could also be used for local/relative importing. Say a project has src/foo.jl and src/bar.jl then in foo.jl:

import bar

would import from src/bar.jl. This is an improvement over using include because it operates within the module system.

[StefanKarpinski]

Yes, packages, package entry points, and modules are expected to be one-to-one. You can break this convention if you need to, but there's not much need for that since modules are just for naming and aren't functional units. The idea that you're bringing up is #4600 except the syntax for a relative import is import .bar whereas import bar is an absolute import.

[trans]

@StefanKarpinski Does import .bar actually look for "bar.jl" in the local directory? I was under the impression that import .Bar referred only to a submodule of an already current parent module.

UPDATE: Duh, that's what you propose in #4600. Sorry.

[ihnorton]

👍. If we are going to do this, 0.4 would be a good time to do it.

[ScottPJones]

👍 Please go ahead and clean some of this up (for 0.4!)

[ssagaert]

instead of having two import mechanisms to distinguish between extending or not, why not signal extension at the extension site itself via a keyword or annotation at the function definition, e.g. override keyword before function? Simillar to @Override annotation in Java.

[ssagaert]

This has the advantage that one clearly sees that the function is overriding another (and hence is a method)

[mbauman]

That's already possible, @ssagaert. You do it by explicitly writing the module name in the function definition (e.g., Base.print(…) = …), and it seems to be a style that lots of folks are converging upon. The only sticking point is that the syntax doesn't work for all possible names (such as .+, etc.).

(As an aside, please be careful to enclose macros with backticks `` to prevent pestering other GitHub users).

[Ismael-VC]

👍 to @ssagaert suggestion of using @override, the original error message when one tries to extend a method without importing it first reads like this:

ERROR: error in method definition: function Foo.x must be explicitly imported to be extended

So pherhaps @extend could be more suited? I'm not a native english speaker, but my understanding is that override means something like: cancel, void, invalidate, negate, annul, nullify, discontinue, etc.

I think this is more explicit:

@extend Base.show(...) = ...

Than:

import Base: show

# ... several lines here

Base.show(...) = ...

[simonster]

@Ismael-VC Base.show(...) = ... already works without importing anything. import is only necessary if you want show(...) = ... to extend Base.show.

[ssagaert]

@Ismael-VC override word was just a suggestion. It can be extend or anything else meaningful. Also there should be no @ since in julia that means its a macro (@Overridereferred to Java where it's an annotation).

[Ismael-VC]

@simonster thank's I didn't know that!

@ssagaert so you mean a keyword? I tried something like this but I still suck at macro-foo:

module Extend

export @extend


macro extend(x)
    mod = x.args[1].args[1].args[1]
    met = x.args[1].args[1].args[2]
    imp = :(Expr(:import, $mod, $met))
    :(Expr(:toplevel, $imp, $(esc(x))))
end


end

I know this is not general, still I can't make an expression that returns what parse does:

julia> using Extend

julia> type Foo end

julia> @extend Base.show(x::Foo) = Foo
:($(Expr(:toplevel, :($(Expr(:import, Base, :show))), show)))

julia> parse("import Base.show; Base.show(x::Foo) = Foo")
:($(Expr(:toplevel, :($(Expr(:import, :Base, :show))), :(Base.show(x::Foo) = begin  # none, line 1:
            Foo
        end))))

I think that a general and working @extend macro wouldn't change the semantics of the import mecanism.

[ssagaert]

@Ismael-VC I did but I like the 'trick' of @mbauman also.

[KristofferC]

Does this being removed from 1.0 mean that we will be left with both using and import for 1.0? That is a bit unfortunate in my opinion.

How about:

• Force module prefix for extension. This will make code clearer that an extension is intended instead of it being accidental from an import in some other file.
• using A becomes import A: ...
• using A.X (X is a module) becomes import A.X: ...
• using A: X (Xis not a module) becomes import A: X
• import A: X is not changed but you cannot automatically extend X (see the first point)
• delete the using keyword

Am I missing some use-case? Maybe this was already suggested...

What I like about being explicit about the module when extending is that extension becomes much more local. Right now, when a method is extended, it is common to put the import very close to the top of the module (which might be in a completely different file!). When the extended method is deleted the import statement is usually forgotten.

[StefanKarpinski]

I think that's essentially the same as my proposal above, which got a fair amount of support. @JeffBezanson really wants to keep using A at least for ease of use and using A: x because apparently (I'm not sold on this), it's an important use case to be able to import a binding in such a way that you cannot extend it. There are some proposals to go the other direction and replace import with using but none of them really got much traction (import seems more fundamental).

[KristofferC]

I think the difference is in:

• import A: x, y # hard bindings for x and y from A

Assuming the meaning of "hard binding" is such that you can extend it without module prefix, in my version there are no hard bindings. If you want to extend, you module prefix it exactly where you extend it. No spooky import statements in other files changing the meaning whether something is an extension or not.

and using A: x because apparently (I'm not sold on this), it's an important use case to be able to import a binding in such a way that you cannot extend it.

Isn't forcing module prefix dealing with that? Or are we talking about non modules like:

module M
    x = 1
end

Both import M: x; x = 2 and using M: x; x = 2 gives the same warning message so I don't see what the problem is there...

Keeping using A for ease of over import A: ... seems a bit excessive in my opinion.

[JeffBezanson]

Isn't forcing module prefix dealing with that?

Yes; if you had to qualify functions to extend them then this point would be irrelevant.

[JeffBezanson]

Keeping using A for ease of over import A: ... seems a bit excessive in my opinion.

I see it the opposite way; making people switch from using A (which is nice and short and we're all quite used to) to import A: ... just to satisfy an artificial requirement that there be only 1 keyword is excessive.

[ggggggggg]

From reading the thread, it seems the main value in having two keywords is to differentiate between bindings that can be extended or not (hard binding). With that in mind, I think there are two viable solutions:

• 1 keyword + require prefix for extending
• two keywords, one for normal (non-extending) use, and a second for extending use I suggest using and extending in this case. import is fine, but extending makes the reason for the existence of a 2nd keyword obvious

In either case I suggest using should be as it is now with the addition of one of the following to bind only the module Foo:

• using Foo: nothing (works now)
• using Foo: Foo (works now)
• using Foo: (can be added later)

Then extending should behave identical to using with the only difference being that you can extend bindings brought in with extending, and possibly disallow extending Foo so that it has to be explicit.

[FelixBenning]

Current behaviour

	make available (using)	make extendable (import)
only module	using module: module or using module: nothing	import module
everything exported	using module (side effect: acts like import module as well)	?
particular things	using module: x,y	import module: x,y

Suggestion

	make available (using)	make extendable (import)
only module	using module	import module
everything exported	using module: *	import module: *
particular things	using module: x,y	import module: x,y

The nice things about this is, that importing more corresponds to writing more. I.e. you start with using module and if you want to import a variable directly into namespace you add a : x instead of removing a nothing or module. It also means that the shortest thing you type includes the least.

You can also do using: *,x to make everything exported available and x which was not exported.

Compromise for backwards compatibility:

	make available (using)	make extendable (import)
only module	using module:	import module:
everything exported	using module: *	import module: *
particular things	using module: x,y	import module: x,y

keep around using module and import module with current behaviour for backwards compatibility but deprecate it.

[tpapp]

@FelixBenning: import Module currently does not (by itself) make anything extendable any more than using Module, it just loads the code and brings Module (and nothing else) into the namespace.

[Seelengrab]

Just to mirror what I've said on slack and so that it doesn't vanish in the slack hole:

I don't think making using X: * the default for making every exported thing available vs. just using X will make people necessarily more wary of what they import. I know, pointing to how others do it is considered bad form, but Python basically has those semantics with their import X and import X: *, yet their ecosystem is littered with those star imports 🤷‍♂️ (and they're famously hating that) I don't think the marginally longer text one has to type stops people from doing what they consider to be the most convenient: just import/use everything and let the compiler figure it out. That's why I'm wary of the magic bullet of making people write that star explicitly.

[mbauman]

Further, import module: * and using module: * is unavailable for the proposed meaning. It already has a meaning as * is a valid Julia identifier and can be imported/used just like + or the word mul.

[FelixBenning]

@tpapp either I have misunderstood the documentation again, or import Module makes Module.x extendable. While using Module: x does not make Module.x extendable. Therefore import Module makes something available for extension and using Module does so too, which is why I noted that using has that side effect.

(from https://docs.julialang.org/en/v1/manual/modules/)

It does not really matter which one of us is right - in either case the current situation is oviously a mess if we can't even figure out what everything does.

@mbauman good point - I forgot about that. I don't really care about the * just the structure of having using mirror the things import does with the difference of import vs using being whether or not things become extendable. So if there is a more appropriate symbol - all, __all__, everything, exported, ...? I am all for it. I just think that importing more should probably be reflected by typing more.

But if you do not want that at all, you could of course also go for

	make available (using)	make extendable (import)
only module	using module: module	import module: module
everything exported	using module	import module
particular things	using module: x,y	import module: x,y

or

	make available (using)	make extendable (import)
only module	using module	import module
everything exported	using module: module	import module: module
particular things	using module: x,y	import module: x,y

But whatever the end result is, it should be consistent. And at the moment it just is not.

[Seelengrab]

using A makes A.f extendable, not f by itself. In order to extend just f without declaring from which module you want it extended, you have to import A: f explicitly. Otherwise you'll still have to qualify it.

Check the following for the semantics of using

julia> module A
        export f
        f() = "no args in A"
       end
Main.A

julia> f()
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] top-level scope at REPL[2]:1

julia> using .A

julia> f()
"no args in A"

julia> f(1)
ERROR: MethodError: no method matching f(::Int64)
Closest candidates are:
  f() at REPL[1]:3
Stacktrace:
 [1] top-level scope at REPL[5]:1

julia> f(x) = "one arg where?"
ERROR: error in method definition: function A.f must be explicitly imported to be extended
Stacktrace:
 [1] top-level scope at none:0
 [2] top-level scope at REPL[6]:1

julia> A.f(x) = "one arg where?"

julia> f(1)
"one arg where?"

And this for the semantics of import:

julia> module A
        export f
        f() = "no args in A"
       end
Main.A

julia> f()
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] top-level scope at REPL[2]:1

julia> import .A

julia> f()
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] top-level scope at REPL[4]:1

julia> A.f()
"no args in A"

julia> f(1)
ERROR: UndefVarError: f not defined
Stacktrace:
 [1] top-level scope at REPL[6]:1

julia> A.f(1)
ERROR: MethodError: no method matching f(::Int64)
Closest candidates are:
  f() at REPL[1]:3
Stacktrace:
 [1] top-level scope at REPL[7]:1

julia> f(x) = "one arg where?"
f (generic function with 1 method)

julia> f(1)
"one arg where?"

julia> A.f(1)
ERROR: MethodError: no method matching f(::Int64)
Closest candidates are:
  f() at REPL[1]:3
Stacktrace:
 [1] top-level scope at REPL[10]:1

julia> A.f(x) = "one arg where in A"

julia> A.f(1)
"one arg where in A"

[tpapp]

@FelixBenning: yes, I think you misunderstood. FWIW, I think that the "... makes Foo.x extendable" is a confusing way to approach the distinction --- you can always define methods to fully qualified function names. What happens with using Foo: x is that Foo itself will not be brought into the namespace.

Incidentally, rereading this topic I am wondering if #25306 brought us to a sort of local optimum, and the only decision that remains is whether we need non-extensible imports into the namespace (currently using Foo: f). But since that is breaking, the manual chapter would perhaps benefit from a rewrite in the meantime, many users find the whole thing confusing.

This is how I would approach the status quo in the docs:

1. using M loads the module, and brings M and its exported symbols into the namespace. This is the only case where exports matter.
2. once you have M, you can use qualified names like M.y to (a) access non-exported symbols and (b) add methods to functions, regardless whether they are exported or not
3. Julia prevents you from adding methods to functions unless you use qualified names like M.f, or ...
4. ... import M: f, then you can just use f when defining methods.
5. import M and using M: x are for bringing just M or x (and not M), respectively, into the namespace, respectively. Some people like to use these forms in package code to make sure their namespaces are kept clean (link relevant style guides here).

The order more or less reflects use cases encountered with more advanced usage. All of the above applies to submodules, with M.A in place of M.

[martinholters]

What about this:

• using M brings M into scope
• using M: x brings M.x into scope (as x)
• using M: ... brings all exported symbols of M into scope
• using M: x, ... brings all exported symbols of M into scope and also x (which might be unexported)
• There is no import. You need to use the qualified name for extending a function. (Or using M; const foo = M.foo like one could already do now.)
• In all of the above, M could also be a submodule, e.g. Foo.Bar and x could also be x as y with the current meaning.

Or we use import instead of using which then makes this equal to #8000 (comment).

[tpapp]

A very common use (especially in "scripts", but also packages for certain styles is)

using Foo, Bar, Baz

and just relying on exported symbols. There would be some benefit from keeping this the simplest, possibly as simple as it is now.

[FelixBenning]

@tpapp

I think you misunderstood. FWIW, I think that the "... makes Foo.x extendable" is a confusing way to approach the distinction --- you can always define methods to fully qualified function names.

Okay, so can I extend Foo.x after using Foo: x? Because if yes, then the documentation is not complete (see my screenshot). If no, then I have perfectly understood how these statements work and quite obviously import Foo did something to make Foo.x extensible. Therefore it literally makes Foo.x available for extension. In every sense of these words. Sure it does not make x available for extension, but that is what import Foo: x is for

[tpapp]

Okay, so can I extend Foo.x after using Foo: x?

Not unless you bring Foo into the namespace somehow (eg using Foo).

I have perfectly understood how these statements work

I am not quite sure about this — however, if you have questions about modules and namespaces, please kindly use the Discourse forum.

quite obviously import Foo did something to make Foo.x extensible

Only in the sense that you have to be able to refer to a function somehow using a qualified name before adding methods to it.

then the documentation is not complete

I think it is, in a kind of quirky way. In that particular example, if all you do is using MyModule: x, p; then no methods are available for extension, so the table is correct.

I do agree that it could be written better, as I said above. A lot of people not used to namespaces find it confusing. And, TBH, the whole using/import circus is mildly confusing, hence this issue.

[FelixBenning]

@tpapp

Okay so here is the thing: It is absolutely not obvious that you can extend every function with the full name if the module is in the namespace. I know that this is current behaviour, but I don't think that anyone who does not know that already would assume that. Especially because being in the namespace does not always mean that it is also extendible. If I do using module:x I can not extend x. While I can extend x, if I use import module:x. Therefore it is a reasonable assumption, that the difference between using and import is whether or not you can extend the the imported functions.

Using this assumption it would make sense, if using module would only allow the use of module.x but would not allow the extension of module.x. While import module would allow the extension of module.x. So if you take this assumption and read the documentation, you will find that two things about that are wrong.

1. using module does allow you to extend module.x. Therefore from a learners prespective using module has the side effect, that it also acts like import module - i.e. it makes module.x extendible. And making things extendible is an import thing, not a using thing
2. in contrast to import, using does not only make module available, but rather everything in it.

So this is what I tried to represent with my table. If two different words are used (import/using) then they should do two different things and that should be clear cut. If using sometimes allows for extension and other times it does not that is not predictable behaviour.

A good alternative is to remove import completely as @martinholters suggests. You just simply can not have two words which are just randomly used for certain things. If import means making things extendible when importing certain functions, while using module: foo does not allow that, then the same behaviour should happen when you only include module in the namespace.

Just because you can extend everything by its full name right now if the module is in the namespace, does not mean that this is an obvious or straightforward thing.

Either being in the namespace is enough, then I should also be able to extend x if I included it in the namespace with using module:x or being in the namespace is not enough, and then I should also not be able to extend module.x when using the using command.
Or third option: there is no import and you simply have to extend functions with their full name all the time.

[tpapp]

It is absolutely not obvious that you can extend every function with the full name if the module is in the namespace.

I agree, which is why I argue for a rewriting of the docs. However, that is tangential to the current issue. It would be best to keep (1) proposals for improving the documentation on the current API and (2) proposals to redesign it separate, even if the two are somewhat related. I plan to make a PR for (1) soon.

[FelixBenning]

@tpapp You can not document something which does inherently not make sense. Neither of these documentations is correct:

1. "You can extend anything which is in the namespace" (because using module:x does not allow you to extend x)
2. "Being in the namespace is not sufficient for extension, this is why separate words using and import exist" (false because for full names it is in fact sufficient to be in the namespace - stopping that to be sufficient was my suggestion)
3. "You can extend anything in the Namespace by its full name" (which would require import module:x to stop existing to be the full truth - @martinholters proposal)

Any of these would be acceptable. But none of these are actually reality. Reality is currently that there are two different words used for "importing stuff" but they don't actually have a distinct use case. It is like a for loop sometimes behaves like a while loop if you iterate over booleans. No amount of documentation will make that non-confusing

[tpapp]

You can not document something which does inherently not make sense.

The current modules API is well-defined, so it can be documented (it already is, I just think it should be better).

Neither of these documentations is correct:

Please be so kind as to wait for my (or someone else's) PR and comment on the actual text that will be there. Positing hypothetical documentation and then claiming that it is incorrect is just adding noise to this discussion.

[FelixBenning]

@tpapp maybe I should have said,

you can not document something in a non confusing way which does not inherently make sense

I mean in a sense the code is all the documentation you need right? Whats wrong with that? The time it takes to digest it. And I currently see no brief way to describe how it works, because it is riddled with exceptions. Exceptions which should not be there in the first place.

Do you really not see what I am trying to convey?

[martinholters]

I think there is general agreement that the current situation is unnecessarily complex and inadequately documented. And no doubt, simplifying the machinery will make documenting it easier. But such simplification will be breaking, so cannot be done before 2.0. So is your point that improvement to the documentation before that wouldn't make sense? Then I disagree. I do see two separate issues: Simplification to be done with 2.0 (which this issue is about) which will (hopefully) include the necessary documentation updates and improving the documentation of the current workings which by reading this thread seems badly required but is another issue.

[tpapp]

After doing #38271, I think that outstanding questions are

How to handle adding methods to functions in other modules

1. always require fully qualified names (Foo.bar() = ...),
2. also just allow when the symbol is in scope (using Foo: bar; bar() = ...)
3. (2), but bring into scope in a special way (status quo, import Foo: bar; bar() = ...)

Which syntax handles export lists and just the module name

1. using Foo brings all exported symbols in Foo to scope, import Foo just the module (status quo)
2. using Foo: ... or some other similar syntax, then using Foo just the module.

(1|2) & 2 would allow unification into a single keyword, either using or import, at the cost of losing single-line

using LinearAlgebra, Random, StaticArrays

I am not sure it is worth it, even without considering the breaking change.

This is not one of those problems that offer a clean solution that the original design just missed; there are trade-offs. I would wait and see if better documentation can improve user experience while keeping the current (1.0) setup.

[bramtayl]

For 2.0 I think just a clean up of the syntax to be more consistent and descriptive of what is actually happening would be nice. Something like:

Before	After
using Foo	useall from Foo
import Foo	use Foo
using Foo: a	use a from Foo
import Foo: a and import Foo.a	extend a from Foo