.types field changed from tuple to SimpleVector

[sebastiang]

Working with reflection-heavy code, the change from v0.3

> immutable X a; b end; X.types
(Any,Any)

to v0.4

> immutable X a; b end; X.types
svec(Any,Any)

is breaking. Granted, it's an internal field. Is there a deprecation/Compat change on the way? It also seems like finding a naked SimpleVector in the wild is perhaps unexpected from my cursory reading of #10380

[JeffBezanson]

SimpleVector behaves like an immutable Any array for all intents and purposes. To keep things simple its use is discouraged, but there's nothing wrong with it. I don't see it as particularly naked or wild.

Yes, this is breaking, but compared to all the breaking changes in 0.4 it barely registers. Given that X.types still implements indexing and iterable interfaces, I don't expect too many problems. Do you have an example of a concrete problem this caused?

[sebastiang]

Minor issues in reflection; see referenced PR. Agreed not too big a deal!

[timholy]

First, let me clarify that I'm all in favor of breaking changes that make julia better. But as one of the main maintainers of HDF5 & JLD, I should publicly admit that I fear we're accumulating a bit of technical debt in terms of our ability to figure out how to load old JLD files on newer versions of julia. Code is something that one can handle with a text editor, but binary data that tries to preserve julia type-formatting is a slightly different kettle of fish. I'm sure it's all solvable with some effort, but it's a category where (1) data files might persist for years and span multiple point-releases of julia, (2) @deprecate does not help (because it's about types and their internal representation), (3) is something for which we do not yet have a great solution, and (4) is a pretty important issue for a technical computing language focused on data.

The same issue is surely relevant to any files that have been saved using julia's serializer.

[sebastiang]

Emboldened by Tim's comment I'll second that notion. Both relying on and breaking internal implementation details is tough sledding. Serialization code, especially 'magic' code that relies on reflection, is always very hard to get right; hats off to the effort expended so far. I think in this case a firmer notion of hiding implementation details would be a good idea from the language core team. But I bet there's been a lot of discussion to date about how and whether to hide . fields, which is what this stems from.

[JeffBezanson]

I'm hopeful that this particular aspect of the change is not hard to support. For example, in Base's deserialize, it only expects the parameters to be saved as something iterable. The Base serializer might even be fully backwards compatible for reading.

[timholy]

After I wrote that comment, I discovered that @simonster has been busy paying off a huge amount of technical debt 😄. I'll advertise his (and @rened's!) impressive contributions in JuliaIO/HDF5.jl#232; most of the changes are due to the tuple overhaul, but there's some other catching up we had to do, too. The main remaining barrier seems to be SubArrays, for which I personally changed the internal representation and am therefore the guilty party.

Really, I think the best way to handle this is break (necessary) stuff as quickly as possible so we get to 1.0 sooner. I just wanted to make sure that folks realize that there's more than code to worry about when we break stuff.

When I think about it, I suspect that keeping up with internal changes is (...perhaps more frustrating but...) not fundamentally different from a different JLD problem we discovered not that long ago: for DataTypes that contain Int fields, saving them on 32 bit machines and loading them on a 64 bit machine and vice versa. For both internal changes and the 32/64 bit problem, you have to do internal type conversion. This kind of problem has nothing to do with julia's rapid evolution, and is the kind of thing that a package like JLD should just expect to have to deal with. (It's on the agenda, like so many other things.)

[timholy]

@sebastiang, the main problem is that JLD aims to be able to save "any" type of julia data (short of things like modules and functions, at least for now) and so at some level needs to look inside of objects. Naturally, if one provides a custom serializer for specific objects, then you can hide internal details. Dicts are a good example where that's already being used.