relative using/import should search current directory

[StefanKarpinski]

When I do

# Main.jl
module Main
  using .Foo
end

and there's a file called Foo.jl in the same directory as Main.jl it should be loaded. I suspect that relative using should also not look in the global require places – i.e. Pkg.dir() and then LOAD_PATH. The same applies to import.

[JeffBezanson]

I agree it should not look in the global places, but perhaps it should not look at files at all. This doesn't seem to extend to multiple dots, e.g. using ..Foo looking in the parent directory.

[StefanKarpinski]

Oops.

[WestleyArgentum]

I would love to see a solution to this problem... But Jeff has a point.

Would be a little weird, but what if using takes a second, optional parameter

using Foo "../../extra-src/"

[tknopp]

+1 for this.

[StefanKarpinski]

I'm not sure the relative path thing is a problem. You could, for example, have something like

# Foo.jl
module Foo
  using ..Bar
  using .Baz
end

# Bar.jl
module Bar
  # barfy stuff
end

# Foo/Baz.jl
module Baz
  # bazish stuff
end

That would allow relative imports of sibling modules to automatically work. Sure, it's strange if you do more dots than you're nested into modules, but just don't do that.

[fxbrain]

Why just not allow to use using with a string?
Such as: using "Baz/Baz.jl".

[StefanKarpinski]

Because loading from a file is only a side effect of using when no such module exists. The normal case is that the module already exists. If you allow a string, you still have to map those strings to a module.

[fxbrain]

But if the string, which is a path, qualifies a module, which it does due to its structure?

[fxbrain]

Ok. Let me try to get this straight.

I do not have an opinion about this "import/include/using" oddity.

Don't you think that sooner or later, the python import strategy turns out to be best?
You guys enabled to have a __init__ function within a module which is automatically called, but not at first, which is quite strange, since I expected it to be called first, just like a BEGIN block in Perl.

I ran into exactly this problem, that I arranged my code in a file directory manner and tried to push! the load-lib path inside the init function. But it is not evaluated at first, which was quite confusing
to me, since I expected it to act such as a CTor or something.

Let's just presume I know what using does semantically. Why not use using to publish several declarations inside the current namespace but give it the ability to assign some prefix for it?

I really liked the import idea of Java. Where dots marked a directory.

For instance:

using Baz.baz

or

using Baz.baz as bz

Does this make sense to you?

Cheers

Stefan

[tkelman]

Bump. Milestoning. #9079 (comment)

[stevengj]

It's a little frustrating to have #12695 merged in 0.4 but this slated for 0.5... I feel like it's going to bite people in 0.4 if there is no way to load modules from the current directory short of modifying the load path.

[JeffBezanson]

We seem to have been getting by pretty well without that --- outside of the REPL, loading from the CWD is just a bug, and I doubt any packages depended on it. In the REPL, include is probably sufficient.

[bermanmaxim]

I used to structure my code into submodules, with each file representing a module ; back when the CWD was included in the load path, this allowed me to use for instance using Utils to load types and functions exported from Utils.jl. I can now replace this with include("Utils.jl"); using .Utils; however, this is inconvenient e.g. if Utils defines types, because creating this type from module A would create an A.Utils.Type instead of a Utils.Type. What is the recommended way of organizing Julia code (with common functions and types) into subfiles ? Should I add the current directory to the path anyway to use the convenience of modules ? Thanks.

[IainNZ]

I have hit the same problem as @bermanmaxim FWIW, and I've moved to just includeing everything instead

[bermanmaxim]

Thanks @IainNZ. Using includes seems indeed to be the standard way now. I guess it's the job of the main file of a module to include everything in the right order to make the subparts work (defining types before functions...) Using distinct modules had the advantage of making the dependencies of each file somewhat more explicit, e.g. putting utils.helperfunction to make clear that the function comes from utils, and not risking including things twice.

[toivoh]

You can use includes in the main file and still structure your code into submodules. That's what I do in Debug.jl.

[mbauman]

I still don't fully understand what this feature should be in the first place. So ignoring the syntax possibilities and just using the quotes to denote this new feature (despite me not liking the quote syntax it's the easiest way to talk about it for now), how should it behave?

In terms of current functionality I think we're all fairly consistently talking about import "File.jl" meaning:

• it implicitly wraps the contents of File.jl in a module File ... end
• it essentially acts like LOAD_PATH was set to the importing file's @__DIR__ and then imports File
• you end up with a module named File in the importing file's namespace

I'm not as clear on using "File.jl". Is it supported? Do you need to export names? Does it automatically grab all names? All names except _leading ones? It seems strange to write export without writing module.

I'm also not clear what this means outside of the context of a single package's implementation. Given a package P that contains a src/File.jl but whose src/P.jl does not explicitly import "File.jl":

• is there a desire to support import P: File?
• is there a desire to support import P; P.File?

Is there desire to detangle julia File.jl from julia -e 'import "File.jl"' in the vein of if __name__ == '__main__'?

[Roger-luo]

it implicitly wraps the contents of File.jl in a module File ... end

this depends on whether import "file.jl" should rejects globals outside module ... end, which means whether do we allow defining foo in file.jl
when it gets import "file.jl":

foo(x) = x

module File
end

• if defining things outside a module ... end is not allowed then this is simple, there is no implicit module needed at all, we can just have using "file.jl": YourModule work or use using YourModule convention.
• if defining things outside a module ... end is allowed, then where does foo belong to?
  • it belongs to the file - we need implicit module for files, this is FromFile.jl
  • it belongs to the package module - we need a way to evaluate these globals from the compiler side, because we will need to evaluate these before other modules

it essentially acts like LOAD_PATH was set to the importing file's @DIR and then imports File

the globals in package module is evaluated at __toplevel__ module, so if we use the first convention, this is true.

you end up with a module named File in the importing file's namespace

depends on which we implement, for the first one, there is no file modules, for the second, import "File.jl" or "using "File.jl" should be exactly what FromFile does currently despite of the syntax.

I'm also not clear what this means outside of the context of a single package's implementation. Given a package P that contains a src/File.jl but whose src/P.jl does not explicitly import "File.jl"

I think P.jl always need to explicitly write either import or using to define the dependency graph, or part of the file dependency relation will be missing

Is there desire to detangle julia File.jl from julia -e 'import "File.jl"' in the vein of if name == 'main'?

if we go with the implicit file module then yes, if not then no.

[PetrKryslUCSD]

Oops, I messed something up: my daughter came into the room and started talking on her phone. My voice recognition picked it up and now there is a message above that I "unassigned" some core devs! Sorry! No clue how to fix it.

[StefanKarpinski]

No problem, I reassigned myself.

[baggepinnen]

I note that your examples have 49 and 41 GitHub stars respectively, and were last updated 5 months ago and 11 months ago. This does very little to engender confidence that they will continue to be maintained.

Assessing stability and whether or not a package will be maintained can not be done by looking at either the number of github stars or when they were last updated. UnPack.jl, as an example, has few stars, but 70 direct dependents and 878 indirect dependents in the general registry. It's an excellent package for what it's doing and will likely remain an excellent package for a long time.
https://juliahub.com/ui/Packages/UnPack/zm2TF/1.0.2?t=2

This is a common occurrence in the Julia ecosystem, small packages that do one thing very well does not need to be updated (no bugs to fix, no features to add), and many may feel they are so trivial and often used as to not warrant a "bookmark star".

[tpapp]

Reflecting on @akbs (sorry, don't know Github username) suggestion on Discourse, this issue could benefit from

1. a package demoing this proposal, insofar that is possible in a package, the feedback from the users.
2. a systematic review of what other languages do, and whether that design makes sense for Julia
3. (discussion of alternative designs is covered in the comments I guess, but more is welcome)
4. suggestions and contributions for tooling and docs that would help address the underlying issue.

[StefanKarpinski]

I very much look forward to this review document and package!

[adkabo]

@StefanKarpinski asked for some people to start using a system of concrete language proposals with this issue as a starting point. I want to pitch in, though I don't have all the background on this issue.

I can write up some things about Python's system since that seems to be the most-discussed implementation here. It would be nice to have some more "foreign language" perspectives.

@Roger-luo would you be willing to turn your understanding into a mini package that captures one of the possible implementations so we can discuss it concretely? Or is there nothing else to add on top of FromFile.jl, that can be done without altering the parser?

I tried to do it myself but afaict it's not possible to write this @importfile using "file.jl" : x as a macro.

[Roger-luo]

@adkabo We have written a proposal and an implementation but it's not people's favorite

IMO the implicit file module is the key of Python module system as far as I understand which is FromFile. If you want to do exactly the same as Python then you only need to let the implicit module name to be the same as file name instead of a mangled name.

so I proposed several alternatives (not all implementable as an external package tho) but no one add thumb up to the alternative I proposed so I assume all these are not favored. I'm not sure if there's actually a revise comment on our proposal, most comments (exclude proposal authors) above seems to just against adding this feature. So I'm also not sure what to revise for the original proposal.

I'll leave it for other people's idea for now unless I have something better in the future.

[mhinsch]

Here is a close to trivial implementation of my suggestion (which is essentially identical to this one made back in 2013). I guess it's a bit impolite to operate on source code instead of Expr trees, but given that the arguments to the using statement do not parse as "regular" Julia code this is by far the easiest way to support the full syntax. Note also that this is completely unchecked and unsafe, it's really just a proof of concept (it is also quite likely that it contains numerous errors - I really only tested the most basic use cases).

using MacroTools

function using_expr(args)
	str = ""
	for a in args
		str *= " " * string(rmlines(a))
	end

	esc(Meta.parse("using" * str))
end

macro rusing(path, args...)
	quote
		pushfirst!($(esc(:LOAD_PATH)), joinpath(@__DIR__, $path))
		$(using_expr(args))
		popfirst!($(esc(:LOAD_PATH)))
	end
end

As you can see it implements a macro @rusing that works exactly like using, but takes a path as an additional first argument. That path gets prepended to the LOAD_PATH relative to the directory of the current file, the corresponding using is executed and then LOAD_PATH is reset to its previous value.
So to load a local module in the same directory:

@rusing "." MyLocalModule

To load a module in a directory "../modules":

@rusing "../modules" SomeOtherModule

The full using syntax is supported, so these should work:

@rusing "." M1, M2
@rusing ".." M3 : function_with_a_long_name as fun1

Needless to say, exactly the same thing could be done for import.

Compared to some of the other suggestions that have been made, this one

• does not interfere with include (which is an orthogonal concept and should stay that way IMHO).
• makes no additional assumptions concerning the correspondence between files and modules (beyond those already made by the current using statement).
• makes no assumptions concerning the layout of source directories. Dump all modules in one directory or do the full Java Christmas tree - it doesn't matter.
• makes one simple and completely transparent change to the language.

For me at least this would solve the one major annoyance of the current module system. I actually wouldn't even mind just using this macro, but unfortunately it's impossible to import macros from a module without language support (AFAICT) so it will always lack functionality.

[einarpersson]

• The current system is a significant pain point for many newcomers.
• The current system makes it hard to explore new codebases...
  Yes. And Yes.

My own experience, coming to Julia with significant experience in a dozen other programming languages, is that the module system was a major pain point for me if not the greatest one. I spent days fighting it and getting confused

In most languages, a file typically constitutes a clear semantic unit and it can be understood on its own in the context of its dependencies, which are listed explicitly (either by explicit import statements or by the use of qualified names).

This is literally me right now (coming from javascript/es6, Go, Python, ...). I really love what I've seen from Julia so far, expect this.

[danielsoutar]

As someone working on a large production codebase (well into 1000s of LOC) in Julia, I can safely say that this issue single-handedly is my biggest gripe with Julia in prod and the reason why I would never use it myself if I had the choice. Huge amounts of code are needlessly recompiled in the tests which defeats the whole philosophy of Julia being a fast and efficient language where you only compile what you need.

While some aspects of the codebase are in theory different/logically separate from each other, in practice they would only ever be used with each other. So the point about having loads of different packages is irrelevant in my case. It would be an unnecessary organizational headache to have to have these as separate packages, either with git sub-repos or possibly with different repos altogether. using "relative-path/to/my-file.jl" for all the reasons explained above is a sensible solution.

I really like the idea of Julia and there are lots of things about the language I like but this massively discourages usage in industry, which discourages usage in general. If you want Julia to be more widely adopted, having the ability to load code once/efficiently from your own package would be helpful. It doesn't force anti-patterns on developers, but forcing people to separate coupled or helper functionality into its own package for the sake of not recompiling it 10s or 100s of times certainly seems like an anti-pattern to me.

[StefanKarpinski]

Huge amounts of code are needlessly recompiled in the tests which defeats the whole philosophy of Julia being a fast and efficient language where you only compile what you need.

What does this issue have to do with repeated compilation?

[danielsoutar]

Huge amounts of code are needlessly recompiled in the tests which defeats the whole philosophy of Julia being a fast and efficient language where you only compile what you need.

What does this issue have to do with repeated compilation?

I'm not sure what you're referring to specifically in that question @StefanKarpinski. If you're asking about the being fast/efficient language where you only compile what you need, then the answer is that it's hard to be fast or efficient in a more holistic sense if the language/loading infrastructure forces you to load internal package code every time you use it, potentially 10s or 100s of times. This is especially true of tests if you write test files as independently runnable files. include can have a place, I just think not having the option to import or using like described above is needlessly limiting.

[pdeffebach]

I think OP is doing

module A
    include("define_base_mod.jl")
end

module B
    include("define_base_mod.jl")
end

which causes re-compilation of methods since the two base_mods are different.

OP please look at FromFile.jl which will enable this workflow without re-compilation. This is not to say there isn't something actionable in this issue though.

[StefanKarpinski]

If that's what you're doing, @danielsoutar, then don't do that—except in unusual situations you should not be including the same code multiple times, you include each file only once. This issue does not affect that. Feel free to ask questions about the right way to use Julia features like include and modules on discourse or slack.

[Roger-luo]

just want to show some numbers here one year later, despite a few people against the proposal I had with @patrick-kidger a while ago in https://github.com/Roger-luo/FromFile.jl and this also answers the question @ChrisRackauckas asked at some point that if a feature really necessary for people, it should be implemented as a package and we can tell that from the number of users.

On JuliaHub the FromFile package had 1.1k new users in past 30 days, and for reference Pluto has 5k new users in past 30 days.

I believe this means something and worth everyone involved in this issue to think further.

[danielsoutar]

I believe this means something and worth everyone involved in this issue to think further.

My two cents: this has been an absolute game-changer for the codebase I'm working on and my team can now flexibly load modules in files where needed without suffering performance penalties. People shouldn't be forced into a C-like project structure from the 1970s where files need to be arranged to minimize includes. That is the wrong level of abstraction to expect developers to work with in 2022.

I define things in some modules, with (optionally) a module per file, and these modules/files should be transparently referenced by the code in every file that uses them. Like helper functions, common types or data. Now I can run individual test files without modification. I seriously encourage people to try this in any non-trivial package and see the benefits for themselves. Thanks to @Roger-luo and @patrick-kidger for this.

[ederag]

At the beginning (few years ago), I also thought that the provenance should be explicit.

Now that I use LSP within kate,
where Ctrl + left clickbrings to the definition, and Ctrl + TAB brings back,
it's much faster than scroll up to see the imports.

And with respect to python, I like the ability to split modules into several files with include (less scrolling).

So it is good that your work style is taken into account, but I also like the current design now.

[alok]

@ederag i feel like you prioritize not scrolling more than many, perhaps most, would.