deserialization problem when cross platforms

[yunqu]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04, Ubuntu 18.04
• Ray installed from (source or binary): source for 18.04, wheels for 16.04
• Ray version: 0.6.5
• Python version: 3.6.5
• Exact command to reproduce:

Describe the problem

When running ray on cross platforms, the serialization/deserialization has problems. The aarch64 platform is running ubuntu 18 while the x86 is running ubuntu 16. I made sure that both have the same python version 3.6.5.

Source code / logs

On X86 side:

python3.6
Python 3.6.5 (default, Apr  8 2019, 15:06:40) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> ray.init(redis_address="x.x.x.x:20000")
>>> @ray.remote(resources={'x86_64': 1}) # i have the ray started with a custom resource
... def add2(a,b): 
...     return a+b
... 
>>> import numpy as np
>>> a = np.array([1,2]) 
>>> b = np.array([3,4])
>>> ray.get(add2.remote(a,b))
array([4, 6])

The above works fine, but if I run the following code on X86 side:

>>> @ray.remote(resources={'aarch64':1})             
... def add2(a,b):
...     return a+b
... 
>>> ray.get(add2.remote(a,b))
terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Fatal Python error: Aborted

Stack (most recent call first):
  File "/usr/local/lib/python3.6/site-packages/ray/worker.py", line 460 in _deserialize_object_from_arrow
  File "/usr/local/lib/python3.6/site-packages/ray/worker.py", line 411 in retrieve_and_deserialize
  File "/usr/local/lib/python3.6/site-packages/ray/worker.py", line 528 in get_object
  File "/usr/local/lib/python3.6/site-packages/ray/worker.py", line 2301 in get
  File "<stdin>", line 1 in <module>
Aborted (core dumped)

[pcmoritz]

This is not supported with the automatic serialization (but we should give a better error message). One way around it is to serialize your data using a library that support s cross-platform serialization (e.g. protobuf, messagepack etc.).

[ddelange]

Does this bug still exist? i.e. a put from arm64 onto an AMD worker, or from amd onto an arm worker? Towards #12128

[richardliaw]

Will close this since it's been too long and we don't plan to automatically support this. Please reopen a new issue if you still want this feature!