Serialization file format for arbitrary julia objects

[jakebolewski]

Python has pretty much standardized on using the .pkl file name extension for serialized python objects. What to call Julia's serialized object dumps? .jld is already taken.

[jiahao]

.jls? (Julia serialized)

[jiahao]

.dat is always good for a round of Twenty Questions.

[Keno]

I think .jls is fine. Otherwise how about .js (JuliaSerialized) ;)?

[JeffBezanson]

+1 for .jls
A related issue is that technically there is no API for saving a .jls "file", only a value at a time. I like the simplicity of this, but it leaves us nowhere to put a header.

[jakebolewski]

Allowing the user to define methods to serialize custom data types also kind of defeats the versioned file concept. The ability to deserialize depends on what is currently loaded into the environment, so the "version" of the format is kind of meaningless.

[JeffBezanson]

We could explore using the serializer in dump.c. That implements the non-overloadable, just-write-the-damn-data kind of serialization. It tends to change rarely, and might be faster too. However it will require modifications not to save the entire heap; e.g. the Base module should just be saved by reference instead of writing all its contents.

[Keno]

Changed the title since the extension seems to be pretty much decided, so the issue now tracks the issue that there is no file format.

[timholy]

I would also argue that if HDF5 can be made essentially as fast (which remains to be seen), we really shouldn't encourage this---there are many advantages in a stable, nearly-human-readable (thanks to h5dump and h5ls), widely-supported format. For that reason, it would be great if rather than saying "wow, HDF5 is slow so let's find something else" if folks threw some effort into investigating & improving its performance.

[simonster]

Like @timholy I think that HDF5/JLD is the way to go unless there is reason to believe it can never be made fast enough.

[timholy]

#7893 now suggests HDF5/JLD can be plenty fast.

[JeffBezanson]

I agree completely --- originally I didn't really want to use serialize/deserialize for anything except message passing, where it is needed to handle details like tracking RemoteRefs and efficiently sending closures around. HDF5/JLD would be much better for persisting data.

[jakebolewski]

How performant is HDF5 on Mac or Windows? Maybe I need to wean myself off past python experience but being able to quickly serailize whatever state you are working on to look at it later is useful. I agree that this is not a good format for "permanent" storage, its more useful for "I'll look at this tomorrow" type storage. I guess if HDF5 is performant enough, and it is easy to install on all platforms then it fills this role quite nicely.

[JeffBezanson]

We could perhaps add save to Base, and use serialize if the extension is jls.

[timholy]

Dunno, I've only tested Linux. I agree with the convenience of serialization, but I've found that for my needs @save "mywholeworkspace.jld" works pretty well. (Usually I pick specific variables, however.) I basically think of .jld files as the equivalent of .mat files.

There are some limitations, though, like not saving functions.

[JeffBezanson]

Saving functions could probably be added by imitating what the code in Base does, but it seems like a fairly marginal need.

[simonster]

After working on JuliaIO/HDF5.jl#132, I'm not sure where to go next here. For @jakebolewski and @jiahao's cases, which involve large arrays of numbers, strings, and immutables, JLD can now beat serialize, but for serializing large arrays of Vector{Any} it's quite pitiful. The problem is that we have to save each object of unknown type as its own HDF5 dataset, which appears to be costly not only (or even mostly) because of disk access but also because of the massive amount of overhead in libhdf5 itself. To get good performance, we're going to have to avoid relying on libhdf5. The question is whether to design our own Julia-specific format or to continue using an HDF5-based format but reimplement the code to read/write it.

I am still of the opinion that the format used by serialize/deserialize is not good as a data storage format. In addition to the instability of the serialize/deserialize format and the fact that it is only meant to save one value at a time, a data storage format should be suitable for archival purposes: It should still be possible to read data from a file even if the types are missing. That doesn't mean serialize/deserialize couldn't be used as a basis for a data storage format, just that more work would be necessary.

[Skylion007]

I'd like second Simonster's point. Unless we fix the issues with JLD, it cannot be a replacement for JLS.

[simonster]

I fixed most of the issues with JLD in JLD2: https://github.com/simonster/JLD2.jl. It reads and writes valid HDF5 files in pure Julia (although for reading it only supports a subset of HDF5). It pretty much works, and it can be many orders of magnitude faster than JLD for the Vector{Any} cases. The caveats: I never tested on Windows; I never got around to implementing compression or groups; HDF5 has substantial file size overhead if you're writing a bunch of small objects (although the performance is typically no slower than 2x Base.serialize, and in some cases it can even be faster). Unfortunately I have to finish my PhD and won't have time to finish it in the near future.

[JeffBezanson]

For 1.0 we should probably add the ability to write a small header giving the format version (and probably an ABI specifier).

[JeffBezanson]

serialize(io, x) and serialize(filename, x) now write a header; I think this can be closed.