[Core] Multi-threaded ray.get can hang in certain situations.

[Catch-Bull]

What happened + What you expected to happen

In actor tasks, the completion of ray.get by any thread can cause all pull requests generated by ray.get in the process (in the pull manager) to be canceled. At this point, other gets will retrigger pull requests, and the progress of the previously pulled objects will be completely reset to zero. This can cause gets to hang directly under the following circumstances.

Versions / Dependencies

ray master: 385e000

Reproduction script

• test.py:

from threading import Thread
import ray; ray.init("auto")

@ray.remote(resources={"worker": 0.1})
class A:
    def run(self, refs):
        ref_largs = refs[0]
        ref_small = ray.put("1")
        def get_small():
            while True:
                ray.get(ref_small)
                import time
                time.sleep(0.1)
        t = Thread(target=get_small, daemon=True)
        t.start()
        print("start get")
        ray.get(ref_largs)
        print("done.")


@ray.remote(resources={"head": 0.1})
def main():
    refs = [ray.put(b"0" * 10 * 1024 ** 3)]
    a = A.remote()

    ray.get(a.run.remote(refs))

ray.get(main.remote())

• run:
• step1: export RAY_object_manager_max_bytes_in_flight=12000000
• step2: ray start --head --resources '{"head": 10}'
• step3: ray start --address='$YOUR_IP:6379' --resources '{"worker": 10}'
• step4: python test.py
  will run forever....

Issue Severity

None

[Catch-Bull]

I'll try to fix it first.

[edoakes]

@Catch-Bull if you can't make progress soon let us know and someone else will pick it up!

[Catch-Bull]

@Catch-Bull if you can't make progress soon let us know and someone else will pick it up!

@edoakes Sorry, I missed this message. I communicated with @jjyao about the fix and just finished implementing it.

[danielgafni]

I suddenly started experiencing ray.get hangs in my Ray Dara code. It’s being invoked during actor initialization. It hangs consistently. It doesn’t hang outside of the actor. I’m not using threads.

Can this be related? Is there a known solution?

I created #56452 to track my issue there in case it's a different one (again, I'm not using multi-threading, but it does happen with distributed Ray only).