conditionally @load non-existent file

[bjarthur]

it would be nice if @load read the file when the code was executed, as opposed to when the macro was expanded, so that errors like that below were not an issue.

julia> using HDF5, JLD

julia> isfile("foo.jld")
false

julia> @load "foo.jld"
ERROR: File foo.jld cannot be found

julia> if(isfile("foo.jld"))
println("foo exists")
@load "foo.jld"
end

ERROR: File foo.jld cannot be found

julia> versioninfo()
Julia Version 0.3.0-prerelease+2486
Commit 7211853 (2014-04-04 06:05 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin13.1.0)
CPU: Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
LAPACK: libopenblas
LIBM: libopenlibm

julia> Pkg.status()
2 required packages:

• HDF5 0.2.20
• PyPlot 1.2.2
  5 additional packages:
• BinDeps 0.2.12
• Color 0.2.9
• Homebrew 0.0.6
• PyCall 0.4.2
• URIParser 0.0.1

[timholy]

If you have any suggestions, I'm all ears. There is no way to:

• Have it create the variables, with the appropriate names, in the calling workspace
• Have it not be a macro

Unlike Matlab, Julia doesn't allow one to assign variables in another workspace.

[timholy]

Hmm, it doesn't look like JLD provides a function to read all variables into a Dict. Would that make you happier? You'd have to extract them manually, of course.

[bjarthur]

i've been trying to come up with a solution and am stumped.

WHY DOES THIS WORK:

julia> macro set_var(var_name,var_value)
sym = esc(symbol(var_name))
ex=:($sym = $var_value)
end

julia> @set_var "foo" 7
7

julia> foo
7

AND NOT THIS:

julia> var_name="bar"
"bar"

julia> var_value=8
8

julia> sym = esc(symbol(var_name))
:($(Expr(:escape, :bar)))

julia> ex=:($sym = $var_value)
:($(Expr(:escape, :bar)) = 8)

julia> eval(ex)
ERROR: syntax: invalid assignment location

[simonster]

esc is only meaningful in a macro; eval has no concept of hygiene. So you'd need:

julia> sym = :bar
:bar

julia> var_value=8
8

julia> ex=:($sym = $var_value)
:(bar = 8)

julia> eval(ex)
8

which indeed works, but doesn't really help here. eval always operates at top level, so bar is always defined in the global scope, even if you execute this code from within a function. Julia won't let you evaluate arbitrary code within a function scope, I think both because it's difficult and because it would deoptimize any function that does this. But it's possible that there will some way to do what you want someday, maybe whenever Julia gets a debugger.

[simonster]

Another possibility is to make this:

@load file begin
    ...
end

Then instead of determining the names of the variables from the contents of the file at compile time, you can determine the variables that are undefined at use in the code block at compile time, and then see if you can read them out of the file at runtime. I'm not sure if this is really worth it, though; the syntax is still kind of ugly.

[bjarthur]

what is the reasoning behind julia not allowing one to assign variables in another workspace?

[simonster]

Julia's functions are quite unlike MATLAB's. In MATLAB, a function scope has an explicitly stored "workspace" that is effectively a Dict{Symbol,Any}. In Julia, running functions really don't have a workspace. At compile time, the compiler finds a place on the stack or a CPU register for every variable inside a given function at compile time, and the running code references those locations. (If it knows the types of the variables, the compiler generates code that stores and operates on the variables more efficiently than it otherwise could.) This is obviously impossible unless you know the variables at compile time (and you know the types can't change).

In principle, one could set things up so that it's possible to dynamically assign to undefined variables that are referenced inside a function, but it wouldn't work with type inference and it's a lot of work for something that probably isn't applicable outside of JLD.

[timholy]

This is the flip side of what makes Julia so much faster than Matlab: you rely on the ability to compile fast code.

[nalimilan]

@simonster's suggestion at #83 (comment) isn't so ugly. It makes sense if you really need to use variables loaded from a file. One could also add typeasserts to allow the compiler to generate efficient code.

Maybe there's a simpler solution: @load could take a function instead of a block, and compile it at run time after loading the file, passing as arguments all the loaded objects. Again, if the caller provides type information about the objects, then the compilation could happen when the macro is evaluated instead.

All of these problems are quite similar to those encountered with DataFrames, where the types of the columns are often not known at compile time. Probably worth comparing the solutions.

[timholy]

I'd like to suggest that this has, for all practical purposes, been closed by @mbauman in #101. Feel free to complain if you disagree.