static compile part 3 (modules)

[vtjnash]

This prepares the serializer to be able to handle incremental loading (aka module caching). Currently there is no interface to it exposed to the user (other than the raw interface I demonstrate below for testing). Static compile part 4 to add the user-friendly interface will be developed shortly, but I wanted to go ahead and get this merged first since that is a separate task. Part 4 is expected to be easy technically but has more UI questions to answer.

julia> @time begin
           FP = ccall(:jl_restore_new_module, Any, (Ptr{Uint8},), "FixedPointNumbers_cache.jlc")
           C = ccall(:jl_restore_new_module, Any, (Ptr{Uint8},), "Color_cache.jlc")
           C2 = ccall(:jl_restore_new_module, Any, (Ptr{Uint8},), "Cairo_cache.jlc")
           G = ccall(:jl_restore_new_module, Any, (Ptr{Uint8},), "Gtk_cache.jlc")
           Gtk.GLib.__init__()
           Gtk.__init__()
       end
elapsed time: 0.691674234 seconds (14420556 bytes allocated)

julia> evalfile(Pkg.dir("FixedPointNumbers/test/runtests.jl"))

julia> evalfile(Pkg.dir("Color/test/runtests.jl"))

julia> evalfile(Pkg.dir("Cairo/test/test_speed.jl"))
<output clipped>

julia> evalfile(Pkg.dir("Gtk/test/tests.jl"))

julia> # All tests passed :)

note: this would have been fully compatible with the current serializer, except that I changed the deser_tag hash-table (with indexes from 0 to 255) into an array.

[ssfrr]

I can't believe nobody's commented on this yet. I am so so excited about this development. After Julia core startup time got so much better it really emphasized how long module load time was. I often avoid restarting my Julia session just so I won't need to reload modules, so this is going to be a huge workflow improvement. Thanks!

[ViralBShah]

This is awesome! It will dramatically improve the JuliaBox experience too.

[dhoegh]

Awesome, I did not know that the development of static compilation of packages where this far. This is going to be a huge selling point for 0.4.

[ViralBShah]

Is it possible to have this backported to 0.3? It's crazy, but thought I'd ask anyways.

[vtjnash]

It's not crazy at all. This was actually developed some months ago in a previous, rejected (and admittedly much less interesting) pull request and I just rebased it recently and then made it a bit more powerful (and thus much more useful). I wouldn't be surprised if it backports cleanly.

but that would require getting this merged soon in 0.4 so that it can start seeing some testing (and to get me to start coding the user interface parts)

[timholy]

I've been occupied by other things and hadn't even noticed this. Really fantastic---certainly one of the most important developments around. I don't really have the expertise to review this, unfortunately, but hopefully we can get this merged soon.

[StefanKarpinski]

So what do you need for this? Testing?

[vtjnash]

unrelated, but perhaps worth noting that if #8008 is merged first, it will probably not be possible to backport this. since that will force me to simplify this PR, thus making it incompatible.

also, it should perhaps be noted here that, at least initially, this will add a strong dependence on the exact sys.ji image, so that any recompilation will force recompilation of all dependencies. thus it is not useful for people developing code in base – although that also means it will be advantageous to think about actually implementing #5155. this would not have been practical a year ago, but with the rapid growth in packages and julia release versions, this is now a sufficient condition to be extremely useful.

@StefanKarpinski testing isn't a bad thing, although since the interface is pretty raw right now, most of the preconditions are tested later by assertions instead making this a little harder to use. I need Jeff's approval for this to actually merge it and start on the next part. The final user interface will look pretty much nothing like the current interface. If anyone wants to try it, the test interface looks something like the following:

using FixedPointNumbers
ccall(:jl_save_new_module, Any, (Ptr{Uint8},), "FixedPointNumbers_cache.jlc", FixedPointNumbers)
ccall(:jl_restore_new_module, Any, (Ptr{Uint8},), "FixedPointNumbers_cache.jlc")
FixedPointNumbers.__init__()

In the final version, the interface will look something like the following:

@inine module FixedPointNumbers # declares that this module may be cached
    import OtherModulesThatAlsoDeclareStaticCompile #declares dependency on OtherM...
    include("other_file_to_include.jl") # declares dependency on other_fi...
end

note: I don't know what to call the macro, but it will behave much like the existing inline macro (adding some metadata to the module Expr), so I've temporarily borrowed the name, to be refined in the PR that actually implements some of that content.

$ ./julia --build $JULIA_HOME/../lib/julia/FixedPointNumbers \
    -J sys.ji ~/.julia/v0.4/FixedPointNumbers/src/FixedPointNumbers.jl

julia> using FixedPointNumbers
    # looks for FixedPointNumbers in path
        # success -> looks in cache for something that matches
            # success -> verify preconditions
   # failure
       # load FixedPointNumbers.jl using above command line, then start again

open question: what preconditions should be validated on the path before loading? weakest is to just always use the cache when available (e.g. *.jlc becomes a valid file extension that Julia will prefer over *.jl when it is found first in the path). this is nice because it add no requirements to the filesystem. it also interacts somewhat nicely with possible future fully static compilation options, in that julia could emit objects with a *.so filename, and seamlessly patch itself together via dlload callbacks.

on the other end of the range of possibilities, it can easily record something about the files that it loaded (timestamp, hash, content) and then decide whether to use the *.jlc file or reject it.

but i've left this question for the very bottom because I don't want it to impact this PR. This open questions has no impact on this PR, and is precisely why I want to separate this into multiple PRs. I suspect I will implement option A, then wrap it in option B as the default, but allowing the user to force the usage of option A where desired.

[timholy]

Since currently we recompile every package each time we load it, having to recompile a package when the julia build has changed is certainly not a substantial barrier. Obviously you don't want to rebuild all packages after finishing a julia build, because that would enormously increase the time needed to build core julia. Just rebuild packages on-demand (related to your last question).

One question: this is module-by-module, not file-by-file? So if I'm working on a big package with ~20 files, a single change to any of them forces a recompile of the entire module? Presuming the answer is "yes, and it would be really hard to implement file-by-file," my suspicion is that developers should be able to split large projects into multiple modules and achieve gains that way. So again no major barrier, I'm just seeking clarification.

Regarding your question at the end about loading the precompiled modules (which I agree is a separate issue): I think we basically have to implement the more complex version. Otherwise people developing packages will be perenially forgetting to delete the old .jlc files and then wondering why their bugfix didn't work. We also have to add a check on the julia build, too (see the first point).

[vtjnash]

yes, julia does scope by the module, not by the file. although, I could patch up vtjnash/Speed.jl to accelerate the line-by-line cache, that only helps if you are editing some module at the end. I suspect doing something with eval(MyModule, :(include("file_to_reload.jl"))) will be the best way to debug this

obviously, yes, you can't rebuild all packages after a rebuild, since you may not even know where they are located. one of my next steps is to handle that.

[StefanKarpinski]

Between #4600 and this, we could maybe just begin to encourage using more submodules when structuring big modules. I'm not sure if that would cut down on this compilation time, but it might.

[vtjnash]

unfortunately, while the serializer work (this PR) generalizes quite well to handling arbitrary submodules, the preconditions I'm am planning using for the next PR don't generalize so easily. that means that it will be relatively trivial to add support for conditional submodule caching, but hard to be more general for embedded submodules.

although, i'm not discouraging #4600

[vtjnash]

Seeing no objections to this, I'll merge and start working on the next part

[timholy]

Glad to hear it!

[StefanKarpinski]

Sweet! What can I do with this?

[jakebolewski]

Do you know how long it takes to generate the cache file for GTK and it's dependencies? Curious about how long Pkg.update will take in the future :-)

[prcastro]

It would be nice to set a flag to a package so it doesn't precompile.

[JeffBezanson]

This PR is missing a description of what the change actually does. All I see is that it prepares us for more stuff in the future, and that it doesn't have an api yet, etc. Ok, but what does it do?

[vtjnash]

Do you know how long it takes to generate the cache file for GTK and it's dependencies? Curious about how long Pkg.update will take in the future :-)

emitting the cache file is pretty negligible in time cost. i didn't measure it directly, however, other than observing that the command jl_save_new_module returns quickly.

This PR is missing a description of what the change actually does. All I see is that it prepares us for more stuff in the future, and that it doesn't have an api yet, etc. Ok, but what does it do?

primarily it makes the "mode" of the serializer more explicit:
https://github.com/JuliaLang/julia/pull/8656/files#diff-669d4cc5c9c8f4573c5f8d57f5dcab20R41

then it uses that mode to enable the creation of another "mode" that is essentially equivalent to MODE_AST (jl_restore_system_image), but which will emit references to parts of the system that are "outside" the thing we are dumping, rather than fully recursing into everything. it then has a final pass to patch up any discrepancies (caused by the uid of types changing). technically, this MODE_AST could instead be implemented using the new mode and passing in jl_base_module, but it is somewhat more efficient to know that the uid of types will be constant afterwards (and that there are no references to outside objects). although as a future project, it would also likely be possible to replace the Base module using this code, thus enabling workspace() to also completely replace Base with a clean copy.

another benefit is that the serializer is now (nearly) reentrant. however, it would require allocating the global state on the stack (

julia/src/dump.c

Line 18 in e3a74ee

static htable_t ser_tag;

) to make it fully reentrant, and I stripped the code out when I was rebasing this pull request

It would be nice to set a flag to a package so it doesn't precompile.

this is likely to be opt-in, at least at first

[JeffBezanson]

I would have used the other argument order --- is child a submodule of parent?

[JeffBezanson]

This change primarily seems to introduce two functions, jl_save_new_module, and jl_restore_new_module. I would expect to see a description of what they do and how they work.

[vtjnash]

this file is just indentation (whitespace) fixes

[vtjnash]

I described how to use them in an earlier comment (#8656 (comment)). However, once the "official" interface is merged, these functions will no longer be DLLEXPORT – that is just for convenience, to make it possible to perform incremental testing at the REPL

[vtjnash]

see bd205a0 for added comments. I'm not sure that I can meaningfully add them here

[timholy]

Do you know how long it takes to generate the cache file for GTK and it's dependencies? Curious about how long Pkg.update will take in the future :-)

emitting the cache file is pretty negligible in time cost. i didn't measure it directly, however, other than observing that the command jl_save_new_module returns quickly.

I suspect this misses the main point of the question. The time to generate the cache file won't be dominated by serialization and I/O, it will be dominated by parsing and lowering all the *.jl files that define the module.

My presumption is that the cache will be regenerated only when the user says using Gtk (either directly or implicitly by using something that requires Gtk), not when Pkg.update() runs. Since currently using Gtk forces us to load all the *.jl files, I'm guessing there should not be much of an extra hit to generate the cache file. On subsequent uses, the load times should be dramatically lower.

So in practice I bet users will eventually learn to become annoyed 😄 by slow responses the first time after a Pkg.update() gives them a new version, but only because they will have become spoiled by the fact that it's fast afterwards.

[timholy]

@vtjnash, what are the odds of using this feature to shorten build times of julia itself? In particular, if someone is working on files that load after osutils.jl, could we shorten build times by defining a Foundation module lying between Core and Base?

[vtjnash]

perhaps: you could either try to set this up to save a Base.Foundation submodule, or setup a slightly different mode in the serializer that gave you "checkpoints" at various points in base.

[JeffBezanson]

Shrinking Base would be good anyway.

[StefanKarpinski]

I'm a bit worried that this may be trying to be too clever and is going to cause a lot of confusion and brittleness when used. Of course, I don't really understand how this is expected to work since there's been no explanation of that provided, just vague indications that code will be compiled and cached.

I suspect that giving the user explicit control will be a bit easier to understand and use: a Base.snapshot(path) function that saves the currently running state as an executable which, when run, starts at the current program state. I know that files and sockets can't be saved like this and I think that's ok – people get that and mostly don't care and we can provide a mechanism for restoring needed state upon startup, perhaps by passing a closure argument to snapshot that will be run after restoration but before continuing.

[vtjnash]

the user can't know what files and state some random library three levels deep might need to restore. this transfers the burden for controlling what code can be cached from the user to the library authors. libraries would need to have a flag to their code that says "Yes, I'll restore any necessary external state in my __init__ method, please arrange to cache me". the users themselves don't need to do anything.

your proposal is also also workable, and roughly equivalent to adding code to the userimg.jl file, and would fail if the user had loaded anything that can't go into that file, such as PyPlot (although, to be fair, it would also not work here, since it saves a number of runtime Ptr{PyObject_struct} values in global variables)

of the two options, i think it is much better for the library authors to be able to control this action than to expect the users to be able to make this decision

Shrinking Base would be good anyway.

yep. it's just an even bigger question then of what moves out, and how. i think we may even want to split base into it's own repository at some point, so that even for binary distributions, we can provide a fully modifiable environment with all the change-tracking and github PR excellence.

[StefanKarpinski]

Could you please write up an email or something where you explain what the strategy here actually is?

[vtjnash]

I don't know what the final strategy will look like in terms of user interactions. ideally, it would be completely transparent to the user and easy for a library author to enable. This is just the framework for experimenting with various proposals. hence also why I wanted to split the technical content here from the PR implementing the documented interface(s). the primary open question is how we should tie modules to files, since currently we don't.
see comments above (#8656 (comment))
or continued discussion / work in #8745

[JeffBezanson]

At this stage the UI is not the issue yet. The issue is the design of the underlying mechanism. For example, an important tidbit I've gleaned so far is that generic functions referenced at the top level of a module will be copied. This implies we are introducing a new operation of "separating" a module that has semantic implications. I know this might not be the final form, but these are exactly the issues we should be discussing.

[vtjnash]

I don't think that occurs very often. However, I could add a
pre-serialization pass that enumerates all toplevel const Functions in all
modules and then serialize them as references instead

[JeffBezanson]

The point is that we need to elucidate and think about all such behaviors --- what other things like that are in here? Do we want to define a notion of what objects are "owned" my a module, and those get serialized and everything else is saved as references? Maybe every generic function should be officially owned by one module or another, i.e. add a module field to MethodTable. Would that help? Etc.

[jakebolewski]

How does this work with caching the output of staged functions? The ArrayView / Generator changes will make them pretty pervasive.

Having a module own a generic function seems like something we should consider. Perhaps it is useful in other in other areas (re-compiling dependent functions)? It seems like the only way to restrict the extensibility of methods in the future (and implement something like Dylan's sealed methods).

[StefanKarpinski]

It's worth keeping in mind how much overlap there is between this and distributed computing. Many of the same issues of ownership and serialization come up in both.

On Oct 21, 2014, at 2:38 PM, Jake Bolewski [email protected] wrote:

How does this work with caching the output of staged functions? The ArrayView / Generator changes will make them pretty pervasive.

Having a module own a generic function seems like something we should consider. Perhaps it is useful in other in other areas (re-compiling dependent functions)? It seems like the only way to restrict the extensibility of methods in the future (and to implement something like Dylan's sealed methods).

—
Reply to this email directly or view it on GitHub.

[jakebolewski]

Along those lines, it would also be great if we could modularize this serializer a bit and reuse much of the same code to tackle some of the performance issues raised in #7893.