[core] Gets timeout on randomly generated ObjectIDs

[stephanie-wang]

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS): 0.8

Normally when retrieving a plasma object, the worker waits until it is sure that the object has been created by contacting the owner and checking whether the task that created the object is still pending. Then, it tries to fetch the object, and if the object still is not available after some timeout, then the object is assumed to have been lost.

For randomly generated object IDs, we do not know what task will create it, so anyone who tries to call ray.get() on the objectID will timeout.

Reproduction (REQUIRED)

import ray
import time

ray.init()

@ray.remote
def fulfill(id):
    time.sleep(11)  # 1s longer than the initial_reconstruction_timeout
    id = id[0]
    ray.worker.global_worker.put_object(None, object_id=id)

random_id = ray.ObjectID.from_random()
fulfill.remote(id)
ray.get(random_id)

Results in:

2020-02-06 09:28:04,280 WARNING worker.py:1511 -- Local object store memory usage:
num clients with quota: 0
quota map size: 0
pinned quota map size: 0
allocated bytes: 1
allocation limit: 2655929548
pinned bytes: 77
(global lru) capacity: 2655929548
(global lru) used: 3.76516e-08%
(global lru) num objects: 1
(global lru) num evictions: 0
(global lru) bytes evicted: 0

Traceback (most recent call last):
  File "test.py", line 14, in <module>
    ray.get(random_id)
  File "/home/swang/ray/python/ray/worker.py", line 1515, in get
    raise value
ray.exceptions.UnreconstructableError: Object 6e991f1c2b7354c8729f5ea57639000000000000 is lost (either LRU evicted or deleted by user) and cannot be reconstructed. Try increasing the object store memory available with ray.init(object_store_memory=<bytes>) or setting object store limits with ray.remote(object_store_memory=<bytes>). See also: https://ray.readthedocs.io/en/latest/memory-management.html

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[stephanie-wang]

cc @ericl @edoakes

[ericl]

I don't think we want to support this feature, could we remove it instead?

[stephanie-wang]

Remove the from_random() API? Yeah, I guess so, but it is useful for testing.