Python patch version compatibility

[sfc-gh-jdu]

Hi, I have a question about using cloudpickle to send objects between two different python patch versions (e.g., 3.8.9 vs. 3.8.10). Will the cloudpickle work for different python patch versions? I'm concerned about that because aligning patch version between server and client is too restrictive for our use case, but README says that "Cloudpickle can only be used to send objects between the exact same version of Python". I've tested that serializing objects in 3.6.8 and deserializing them in 3.6.12 works, but I'm not sure it will work for all Python patch versions.

Could anyone answers this question? Thanks!

[pcmoritz]

It would be interesting to find out if the cloudpickle protocol is stable between patch versions of Python :)

Judging by the current README it seems like the answer is "no" (it mentions the exact version of python needs to match).

[shobsi]

Maybe it is due to the same (major, minor) version between 3.6.8 and 3.6.12? I can repro that cross version doesn't work. My code is as below:

Serialization:

import cloudpickle

def add_one(x):
    return x + 1

with open('add_one.cloudpickle', "wb") as f:
    cloudpickle.dump(def_, f, protocol=4)

Deserialization:

import cloudpickle

with open("add_one.cloudpickle", "rb") as f:
  add_one = cloudpickle.load(f)

if __name__ == "__main__":
  print("I am in main")

I serializaed in python version 3.10.x.

Deserialization in python 3.10.x:

$ python main.py
I am in main
$

Deserialization in python 3.11.y:

$ python main.py
Traceback (most recent call last):
  File "/usr/shobsi/cloudpickle-test/main.py", line 9, in <module>
    add_one = cloudpickle.load(f)
TypeError: code expected at most 16 arguments, got 18
$

[HyukjinKwon]

IIRC pickle isn't compatible even between minor versions. This is the limitation of pickle itself.

Also, README.md mentions:

Cloudpickle can only be used to send objects between the exact same version of Python.

[cwiede]

We also stumbled over this limitation and I wonder what is the root cause of this? I clearly understand that a compatibility across python minor versions might be difficult because of python bytecode incompatibility.

But as far as I understand (see e.g. https://peps.python.org/pep-3147/) python bytecode is compatible across patch versions of python, so the question is if there is something else preventing compatibility across python patch versions (e.g. the above mentioned 3.6.8 and 3.6.12)?

If there is no reason for incompatibility, then maybe an update of the readme would be reasonable, so that depending projects may relax their version checks (see ray-project/ray#3988).

[jjyao]

Anyone from cloudpickle can answer the above question? cc @ogrisel