Make TransactionMetaData in charge of (de)serializing extension data

[jmuchemb]

IStorage implementations used to do this task themselves which leads to code
duplication and sometimes bugs (one was fixed recently in NEO). Like for object
serialization, this should be done by the upper layer (Connection).

This commit also provides a way to get raw extensions data while iterating
over transactions (this is actually the original purpose[2]). So far, extension
data could only be retrieved unpickled, which caused several problems:

• tools like zodb dump [1] cannot dump data exactly as stored on a
  storage. This makes database potentially not bit-to-bit identical to
  its original after restoring from such dump.

• zodb dump output could be changing from run to run on the same
  database. This comes from the fact that e.g. python dictionaries are
  unordered and so when pickling a dict back to bytes the result could
  be not the same as original.

  ( this problem can be worked-around partly to work reliably for e.g.
  dict with str keys - by always emitting items in key sorted order,
  but it is hard to make it work reliably for arbitrary types )

Both issues make it hard to verify integrity of database at the lowest
possible level after restoration, and make it hard to verify bit-to-bit
compatibility with non-python ZODB implementations.

For this, TransactionMetaData gets a new 'extension_bytes' attribute and
and common usage becomes:

• Application committing a transaction:

  • 'extension' is set with a dictionary
  • the storage gets the bytes via 'extension_bytes'

• Iteration:

  • the storage passes bytes as 'extension' parameter of TransactionMetaData
  • the application can get extension data either as bytes ('extension_bytes')
    or deserialized ('extension'): in the former case, no deserialization
    happens and the returned value is exactly what was passed by the storage

[1] https://lab.nexedi.com/nexedi/zodbtools
[2] #183

Co-Authored-By: Kirill Smelkov [email protected]

[jmuchemb]

I recently got more interested by #183 while doing 2 things in NEO: check that a FS DB was correctly imported (using zodbtools), and fight with bugs and code duplication about serialization of extension data.

But like Jim, I prefer to serialize between extension and the new extension_bytes when necessary. I do it lazily with properties.

Since extension_bytes is not optional anymore, many comments in #183 become irrelevant and it looks less controversial to define a new attribute IStorageTransactionMetaData, so I still do it.

[jamadden]

I think I like this (though I'm not fond of the magic that the decorator uses to get the attribute names from the argspec)!

[jmuchemb]

though I'm not fond of the magic that the decorator uses to get the attribute names from the argspec

I found important to minimize code duplication here, so that both properties remain always symmetric. I wish I could find something better. Despite code that is a little harder to read, there's still little some duplication at the places _extension_property is used.

I don't want to wait too long before merging this one, as I have other changes I'd like to submit.

I plan to merge mid-September, unless anyone wants more time to review.

[icemac]

What is the status of this PR? What needs to be done to get it merged?

[jmuchemb]

First, I rebased the original commit. But even with the small diff I suggested above, it would remain possible to mutate extensions after having kept a reference to them.

Magic does not look a good idea if the result is not perfect, especially when there's no need.

So I changed for something simpler, following tseaver comment partially. See IStorageTransactionMetaData.__doc__ for more details.

[jmuchemb]

I'll merge next monday.