-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: fork py38 / py312 pickle C extensions #92
base: master
Are you sure you want to change the base?
Conversation
We need to be able to build this on other versions of Python, and without having the original sources available.
Attempt to conform to the new argument-clinic patterns present in the forked modules.
Pass the unpickler's statck.
Add comments annotating semantics of all 'load_*' opcodes handlers shadowed by 'noload_' opcode handlers, and the corresponding 'noload_' semantics.
Use current (Python 3.8) 'Unpickler.load' as a template, leaving the 'load_*' opcodes commented out for eyeball verification.
The Python '_Py_ID' function is not accessible outside its own build.
Most changes involve finding / passing the 'PickleState' pointer into 'noload' and the 'noload_*' opcode handlers. 'Unpickler.noload' is now declared using the required convention to be able to look up module state from the type.
Databases created under Python 2 should have been migrated to Python 3 using https://pypi.org/project/zodbupdate/ – at least this is what we have promoted to do to upgrade existing applications to Python 3. |
Hello everyone, For the reference: in the context of ERP5 we do load pickles generated on py2 under py3 instead of migrating the database. Quoting https://lab.nexedi.com/nexedi/pygolang/-/merge_requests/21#note_205356 :
From this point of view it would be handy to retain support for STRING opcodes. Thanks beforehand, P.S. Please see https://lab.nexedi.com/nexedi/pygolang/-/merge_requests/21 for the whole context of this if you are interested. |
Interesting: I'd forgotten that tool existed.
I'd be glad to just |
I just looked, and the
For writing, the new modules lose the ability to emit the
Examining the |
@tseaver, thanks for feedback. Maybe when originally replying I misunderstood a bit the following:
I understand and agree that all _pickle_33, _pickle_38 and _pickle_312 handle For the reference I remembered that my Your outline of how loading works and how saving does not work for STRING However on my side loading and saving https://lab.nexedi.com/kirr/pygolang/-/blob/58f86b12/golang/_golang_str_pickle.pyx#L374-393 while for saving the code is adjusted to emit https://lab.nexedi.com/kirr/pygolang/-/blob/58f86b12/golang/_golang_str_pickle.pyx#L395-423 This way what py2 saved as its In total there are 3 types in this scheme:
The binary string and unicode string are just two different representations of And for the reference: even though intended encoding for binary string My primary message here is that there still are users that care about Kirill |
Build either '_pickle_38.c' or '_pickle_312.c', based on Python version.
Python < 3.12 uses the version from Python 3.8; Python >= 3.12 uses the version from 3.12. Note: this change does *not* yet fix up all the things in the forked modules: that bit is still in progress.
New features:
Remaining issues:
_pickle_33.c
, and build only one extension module (much simpler). If so, users who require the older "preserve pickle-level compatibility for databases created under Python 2" scenario will need to pin to the last major version, forever.setup.py
to build the desired extension, based on Python version.