Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide a way to retrieve raw extensions data #183

Closed
wants to merge 1 commit into from
Closed

Provide a way to retrieve raw extensions data #183

wants to merge 1 commit into from

Commits on Apr 5, 2018

  1. Provide a way to retrieve raw extensions data 

    Currently when client has IStorageTransactionMetaData instance (e.g. on
storage iteration) it is possible to read transaction's .user and
.description in raw form, but .extension is always returned unpickled.

This creates several problems:

- tools like `zodb dump` [1] cannot dump data exactly as stored on a
  storage. This makes database potentially not bit-to-bit identical to
  its original after restoring from such dump.

- `zodb dump` output could be changing from run to run on the same
  database. This comes from the fact that e.g. python dictionaries are
  unordered and so when pickling a dict back to bytes the result could
  be not the same as original.

  ( this problem can be worked-around partly to work reliably for e.g.
    dict with str keys - by always emitting items in key sorted order,
    but it is hard to make it work reliably for arbitrary types )

Both issues make it hard to verify integrity of database at the lowest
possible level after restoration, and make it hard to verify bit-to-bit
compatibility with non-python ZODB implementations.

To fix we provide a way to retrieve raw extension from transaction
metadata. This is done in backward-compatible way by introducing
new IStorageTransactionMetaDataRaw interface, which may be optionally
provided by an IStorageTransactionMetaData instance, and if so there
will be additional

	.extension_bytes

attribute.

Then ZODB.Connection.TransactionMetaData (the only in-tree
IStorageTransactionMetaData implementer) constructor is extended to see
whether it was given extension in raw form, and if yes, remember it as
.extension_bytes in addition to deserialized .extension dict.

This way storages that now pass raw extension to TransactionMetaData will
automatically provide .extension_bytes in their iterated transactions.

Teach FileStorage & friends to provide/use new functionality.

/helped-by: Jim Fulton.

[1] https://lab.nexedi.com/nexedi/zodbtools
    @navytux
    navytux committed Apr 5, 2018
    Configuration menu
    Copy the full SHA
    7f5a67b View commit details
    Browse the repository at this point in the history