Can I ignore python version mismatch? #3988
Comments
|
Also I wonder if using 3.5.2 and 3.5.6 causes same error |
|
This might lead to trouble, since two different versions of pickle might produce incompatible serialized objects. I can understand if you can't upgrade the OS, but would it be possible to use Ray on Anaconda Python and install the same anaconda version on both machines? You can even install anaconda without root access by the way. |
|
Thank you. Anaconda works well! |
|
Sorry to comment on this, but I do think the version mismatch check is too harsh... Having versions like 3.6.5 and 3.6.7 will lead to an error, but I guess the infrastructure should just work fine. For me, I just modified if version_info[:2] != true_version_info[:2]:
#raise Exception(error_message)
passIt looks fine to me. Maybe that error message should be a warning instead? |
|
Hey just came across this issue. At the very least Ray should be able tolerate differing patch versions of Python between nodes. I'm running into a bit of a dependency headache caused by setting up gpu type nodes and cpu type nodes in Docker. The two types start of with different base Docker images and it's a bit difficult to get them both to have the same Python patch versions. |
|
Can we reopen this? Having the same issue as @mattdornfeld |
|
Just chiming in here, we've also encountered this check, and found it to be too strict. Especially difficult to run Ray cluster with different base images. |
|
Actually I don't think this can be supported. In the cloudpickle README it states that The reason for that is cloudpickle might e.g. pickle lambdas which might include Python bytecode, which can change even between minor Python versions. Now it might work in practice (and probably it will in many cases), but it is a dangerous game to play and might lead to subtle bugs. If you want to take the risk and e.g. have an environment variable/flag to ignore the check, that's something we can consider. |
|
@pcmoritz Agreed about different minor versions, but what about relaxing this check for the patch versions? (i.e. 3.9.6 and 3.9.7) |
|
Related: cloudpipe/cloudpickle#451 |
|
With "exact Python version" they probably mean everything, including patch but let's see about the discussion in the cloudpickle issue. |
|
Hi, I'm a bot from the Ray team :) To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months. If there is no further activity in the 14 days, the issue will be closed!
You can always ask for help on our discussion forum or Ray's public slack channel. |
stale
bot
added
the
stale
|
Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message. Please feel free to reopen or open a new issue if you'd still like it to be addressed. Again, you can always ask for help on our discussion forum or Ray's public slack channel. Thanks again for opening the issue! |
As discussed in ray-project#3988, the intention of check_version_info is to ensure that cloudpickle can transfer objects across the two Python interpreters. Because cloudpickle serializes Python bytecode, it is known to be incompatible across minor versions (e.g., 3.11 to 3.12) but should be compatible across patch versions (e.g., 3.12.0 to 3.12.1). The intention of the CPython team is that bytecode should be compatible within patch releases to the same minor version. This was last broken in 3.5.3, which was regarded as a mistake: see the commit message of python/cpython@93602e3 (in 3.5.10), which implies that it is reasonable to "rel[y] on pre-cached bytecode remaining valid across maintenance releases." The constant importlib.util.MAGIC_NUMBER is used by Python to identify bytecode compatibility (it is stored in .pyc files and checked when they are loaded). The unwanted change in 3.5.3 bumped MAGIC_NUMBER, but since then, it has only changed between minor versions. See the long comment in CPython's Lib/importlib/_bootstrap_external.py for a history of the changes. So, the practical effect of checking MAGIC_NUMBER will generally be to enforce that nodes run the same minor version of Python, but if some future patch release unexpectedly does break bytecode compatibility, checking MAGIC_NUMBER will account for that.
As discussed in ray-project#3988, the intention of check_version_info is to ensure that cloudpickle can transfer objects across the two Python interpreters. Because cloudpickle serializes Python bytecode, it is known to be incompatible across minor versions (e.g., 3.11 to 3.12) but should be compatible across patch versions (e.g., 3.12.0 to 3.12.1). The intention of the CPython team is that bytecode should be compatible within patch releases to the same minor version. This was last broken in 3.5.3, which was regarded as a mistake: see the commit message of python/cpython@93602e3 (in 3.5.10), which implies that it is reasonable to "rel[y] on pre-cached bytecode remaining valid across maintenance releases." The constant importlib.util.MAGIC_NUMBER is used by Python to identify bytecode compatibility (it is stored in .pyc files and checked when they are loaded). The unwanted change in 3.5.3 bumped MAGIC_NUMBER, but since then, it has only changed between minor versions. See the long comment in CPython's Lib/importlib/_bootstrap_external.py for a history of the changes. So, the practical effect of checking MAGIC_NUMBER will generally be to enforce that nodes run the same minor version of Python, but if some future patch release unexpectedly does break bytecode compatibility, checking MAGIC_NUMBER will account for that.
As discussed in ray-project#3988, the intention of check_version_info is to ensure that cloudpickle can transfer objects across the two Python interpreters. Because cloudpickle serializes Python bytecode, it is known to be incompatible across minor versions (e.g., 3.11 to 3.12) but should be compatible across patch versions (e.g., 3.12.0 to 3.12.1). The intention of the CPython team is that bytecode should be compatible within patch releases to the same minor version. This was last broken in 3.5.3, which was regarded as a mistake: see the commit message of python/cpython@93602e3 (in 3.5.10), which implies that it is reasonable to "rel[y] on pre-cached bytecode remaining valid across maintenance releases." The constant importlib.util.MAGIC_NUMBER is used by Python to identify bytecode compatibility (it is stored in .pyc files and checked when they are loaded). The unwanted change in 3.5.3 bumped MAGIC_NUMBER, but since then, it has only changed between minor versions. See the long comment in CPython's Lib/importlib/_bootstrap_external.py for a history of the changes. So, the practical effect of checking MAGIC_NUMBER will generally be to enforce that nodes run the same minor version of Python, but if some future patch release unexpectedly does break bytecode compatibility, checking MAGIC_NUMBER will account for that.
As discussed in ray-project#3988, the intention of check_version_info is to ensure that cloudpickle can transfer objects across the two Python interpreters. Because cloudpickle serializes Python bytecode, it is known to be incompatible across minor versions (e.g., 3.11 to 3.12) but should be compatible across patch versions (e.g., 3.12.0 to 3.12.1). The intention of the CPython team is that bytecode should be compatible within patch releases to the same minor version. This was last broken in 3.5.3, which was regarded as a mistake: see the commit message of python/cpython@93602e3 (in 3.5.10), which implies that it is reasonable to "rel[y] on pre-cached bytecode remaining valid across maintenance releases." The constant importlib.util.MAGIC_NUMBER is used by Python to identify bytecode compatibility (it is stored in .pyc files and checked when they are loaded). The unwanted change in 3.5.3 bumped MAGIC_NUMBER, but since then, it has only changed between minor versions. See the long comment in CPython's Lib/importlib/_bootstrap_external.py for a history of the changes. So, the practical effect of checking MAGIC_NUMBER will generally be to enforce that nodes run the same minor version of Python, but if some future patch release unexpectedly does break bytecode compatibility, checking MAGIC_NUMBER will account for that. Signed-off-by: Geoffrey Thomas <[email protected]>
As discussed in ray-project#3988, the intention of check_version_info is to ensure that cloudpickle can transfer objects across the two Python interpreters. Because cloudpickle serializes Python bytecode, it is known to be incompatible across minor versions (e.g., 3.11 to 3.12) but should be compatible across patch versions (e.g., 3.12.0 to 3.12.1). The intention of the CPython team is that bytecode should be compatible within patch releases to the same minor version. This was last broken in 3.5.3, which was regarded as a mistake: see the commit message of python/cpython@93602e3 (in 3.5.10), which implies that it is reasonable to "rel[y] on pre-cached bytecode remaining valid across maintenance releases." The constant importlib.util.MAGIC_NUMBER is used by Python to identify bytecode compatibility (it is stored in .pyc files and checked when they are loaded). The unwanted change in 3.5.3 bumped MAGIC_NUMBER, but since then, it has only changed between minor versions. See the long comment in CPython's Lib/importlib/_bootstrap_external.py for a history of the changes. So, the practical effect of checking MAGIC_NUMBER will generally be to enforce that nodes run the same minor version of Python, but if some future patch release unexpectedly does break bytecode compatibility, checking MAGIC_NUMBER will account for that. Signed-off-by: Geoffrey Thomas <[email protected]>
As discussed in ray-project#3988, the intention of check_version_info is to ensure that cloudpickle can transfer objects across the two Python interpreters. Because cloudpickle serializes Python bytecode, it is known to be incompatible across minor versions (e.g., 3.11 to 3.12) but should be compatible across patch versions (e.g., 3.12.0 to 3.12.1). The intention of the CPython team is that bytecode should be compatible within patch releases to the same minor version. This was last broken in 3.5.3, which was regarded as a mistake: see the commit message of python/cpython@93602e3 (in 3.5.10), which implies that it is reasonable to "rel[y] on pre-cached bytecode remaining valid across maintenance releases." The constant importlib.util.MAGIC_NUMBER is used by Python to identify bytecode compatibility (it is stored in .pyc files and checked when they are loaded). The unwanted change in 3.5.3 bumped MAGIC_NUMBER, but since then, it has only changed between minor versions. See the long comment in CPython's Lib/importlib/_bootstrap_external.py for a history of the changes. So, the practical effect of checking MAGIC_NUMBER will generally be to enforce that nodes run the same minor version of Python, but if some future patch release unexpectedly does break bytecode compatibility, checking MAGIC_NUMBER will account for that. Signed-off-by: Geoffrey Thomas <[email protected]>
System information
Describe the problem
Hi!
I'm trying to make cluster between ubuntu 16.04 and 18.04 machines.
I just realized that two OS have different default python3 version (3.5.2 and 3.6.7) after installing everything.
And now ray says it can't connect because of python version mismatch
I assume that there are no big difference between 3.5.2 and 3.6.7 for running my codes so
I want to ignore the mismatching. Is there any way to do this?
Or is it impossible because of some ray internal mechanisms?
Any help will be appreciated. Thanks!
Source code / logs
The text was updated successfully, but these errors were encountered: