Reconstructing types defined in one module inside another

[qua4tre]

The following worked in julia v0.6 but no longer works in v0.7:

I have a type defined in ModA.jl (in LOAD_PATH):

module ModA

struct MyType
    a::Int64
    b::Float64
end

end # module

I save data with my type in a JLD2 file:

julia> import ModA
[ Info: Precompiling ModA [top-level]

julia> using FileIO

julia> t = ModA.MyType(1, 0.0)
ModA.MyType(1, 0.0)

julia> save("t.jld2", "t", t)

In some code I run separately, I try to load the data inside a module and I get a warning:

julia> module ModB

       import ModA
       using FileIO

       t = load("t.jld2", "t")

       end
┌ Warning: type ModA.MyType does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/xpVxm/src/data.jl:1153
Main.ModB

julia> ModB.t
getfield(JLD2.ReconstructedTypes, Symbol("##ModA.MyType#368"))(1, 0.0)

Then if I try to interact with ModB.t (e.g. call functions on it defined only on type MyType), I get an error (because the type is wrong). This is similar to #82; may be related?

The problem is alleviated by doing an import ModA at the top level before I try to load ModB, so I suspect the reason this worked on v0.6 and doesn't on v0.7 is due to (JuliaLang/julia#17997). I'm wondering if this just means I need to change my workflow and always load modules with user-defined types at the top level before trying to load files containing those types, or if this is something that can be addressed in JLD2? For example, would there be a way to make JLD2 "see" that ModA has been imported inside ModB and recognize ModA.MyType without the need to explicitly import ModA at the top level?

[xiuliren]

I got a similar issue. this blocked my work!

[MoonCoral]

This screwed me over to.
The behaviour I would personally like to see is:

• to first try to to import missing modules
• failing that not to blindly create a new data type but to store the data in a JLD2 type such
  • that if it is written back to a jld2 file as if it had been properly loaded
    • that is, no risk of loss or corruption from a load save cycle
    • a use case would be to concatenate two jld2 files produced by a distributed process
  • that the data can be manually recovered in an interactive environment, similar to the current environment

[BoundaryValueProblems]

Yes, I have the same problem! Interestingly, the current version of JLD v0.9.0 works without any problem of saving and loading such composite data types under Julia v1.0.1 although the saved file size is larger than that of JLD2.

[richiejp]

This might be related to #126

You can call JLD2.typename on a type to see what fully qualified name is being used.

[kleinschmidt]

This happens for me even for types defined in packages that are on the load path/project:

julia> module Sandbox3
           using DataFrames, JLD2
           x = DataFrame(a = 1:10)
           @save "x.jld2" x
           @load "x.jld2"
       end
[ Info: Recompiling stale cache file /home/dave/.julia/compiled/v1.0/JLD2/O1EyT.ji for JLD2 [033835bb-8acc-5ee8-8aae-3f567f8a3819]
┌ Warning: type DataFrames.DataFrame does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/dev/JLD2/src/data.jl:1143
┌ Warning: type DataFrames.Index does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/dev/JLD2/src/data.jl:1143
Main.Sandbox3

[kleinschmidt]

Which makes it basically impossible to use JLD2 in Weave.jl (at least without turning sandboxing off which @evals all the code in Main) (JunoLab/Weave.jl#164)

[photicus]

I'm seeing the same issue in trying to upgrade some code from Julia 0.6. What seems to work is to have all the necessary packages loaded in the Main module. When I run what @kleinschmidt in the REPL I see the same issue. However, if I put in using DataFrames before defining the module it works just fine. I'm note sure why this works though...

This isn't a viable workaround in the long run, but hopefully it sheds some light on what's going on.

[qua4tre]

It seems JLD2 looks for modules in Main only and can't see modules imported inside the current module.

A related workaround I found (can't remember where right now) is to add a line Core.eval(Main, :(import MyMod)) for each module that defines types that I need JLD2 to be able to recognize.

[photicus]

So I tried seeing what happens if I tried using JLD instead of JLD2. The code above still performes a reconstruction warning (not the case in Julia0.6). However, with JLD there's the addrequire function which provides automatic loading of modules into Main. I don't know if there's something similar for JLD2.

[AmebaBrain]

It seems JLD2 looks for modules in Main only and can't see modules imported inside the current module.

A related workaround I found (can't remember where right now) is to add a line Core.eval(Main, :(import MyMod)) for each module that defines types that I need JLD2 to be able to recognize.

Probably it was here :)

Adding Core.eval(Main, :(using SomeModule)) into your module __init__ function allows you to automatically push all required modules definition into the Main module scope during execution of using MyModule and thus make issue resolved

[timholy]

This should have been fixed by #151, #164, #166, and #173, which are part of 0.1.11. If you're using this version and still encountering problems, please file a new issue.