support different byte orders in serializer

[vtjnash]

In the serializer (and elsewhere?), array reads and writes probably should be done in network byte order.

[StefanKarpinski]

God no. Network byte order needs to die. There's nothing better about it. If anything, the serializer should have a BOM in the header of the data stream and conditionally translate from the serialized order, which ought to be native to the sender, to the native order of the receiver. Alternatively, the receiver can just die if the BOM doesn't match its local endianness. Who the hell runs a cluster with heterogenous endianness? That's a silly thing to do.

[vtjnash]

It's better only in that it is consistant. Isn't Serialization used for writing files too? Adding a header with endianness makes sense too (since it reduces work for the common case of little-endian to little-endian). I was talking with Jeff about making a few changes to the structure of the serialization protocol, to make it more robust against other sorts of data corruption issues.

[StefanKarpinski]

My general opinion on endianness and serialization formats is that byte swapping should be done lazily even for writing files because by far the most common case is that you are communicating between machines with the same endianness. Moreover, these days it is almost certainly little-endian, especially since Julia doesn't even run on any big-endian architectures. Are there any common big-endian architectures that people even use these days?

[vtjnash]

PowerPC (the only other architecture currently supported by the LLVM JIT) is big/bi-endian. It shows up in FPGAs (e.g. Xilinx Virtex 4/5), all modern game consoles (e.g. XBox360, Wii and PS3 are big-endian), high-performance computers[citation needed], and old apple computers (which are stuck at Mac OS 10.5).

ARM is little/bi-endian. So it can be used in big-endian mode (although it appears that Android generally uses little-endian everywhere).

[StefanKarpinski]

Ok, well, it sounds like something we ought to plan to accommodate, but not at the expense of making the vastly most common case of little-endian-to-little-endian do lots of pointless byte swapping.