[Core] AttributeError: 'Worker' object has no attribute 'import_thread'

[MissiontoMars]

What happened + What you expected to happen

2024-03-14 21:22:03,766	WARNING worker.py:2067 -- Traceback (most recent call last):
  File "python/ray/_raylet.pyx", line 1807, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 1670, in ray._raylet.execute_task_with_cancellation_handler
  File "/usr/local/lib/python3.9/dist-packages/ray/_private/function_manager.py", line 394, in get_execution_info
    self._wait_for_function(function_descriptor, job_id)
  File "/usr/local/lib/python3.9/dist-packages/ray/_private/function_manager.py", line 483, in _wait_for_function
    self._worker.import_thread._do_importing()
AttributeError: 'Worker' object has no attribute 'import_thread'
An unexpected internal error occurred while the worker was executing a task.

The import thread has been removed #36293, and it appears that the code within function_manager should also be deleted.

Versions / Dependencies

ray2.6.1

Reproduction script

None

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[MissiontoMars]

@scv119 import_thread in function_manager should be removed?

[jfaust]

Is the fix for this in 2.10? It looks like it might have merged in time (Mar 15, vs. Mar 21 release date).

I'm running into this while trying to find a repro for #44438, with this script:

import ray
import numpy as np

ray.init()


@ray.remote
def _create_data():
    return np.zeros((10000, 100))


@ray.remote
def _mul(data):
    return np.multiply(data, data)


@ray.remote
def _test(data):

    ray.get(_mul.remote(data))


@ray.remote
def _toplevel():
    tasks = {i: _test.remote(_create_data.remote()) for i in range(0, 100)}
    return tasks


results = ray.get(_toplevel.remote())

If I run it a few times in a row I'll eventually get:

2024-04-03 10:08:28,301 INFO worker.py:1567 -- Connecting to existing Ray cluster at address: 172.23.208.127:6379...
2024-04-03 10:08:28,315 INFO worker.py:1743 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265 
(raylet) Traceback (most recent call last):
  File "python/ray/_raylet.pyx", line 2281, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 2161, in ray._raylet.execute_task_with_cancellation_handler
  File "/home/josh/work/.../venv/lib/python3.8/site-packages/ray/_private/function_manager.py", line 391, in get_execution_info
    self._wait_for_function(function_descriptor, job_id)
  File "/home/josh/work/.../venv/lib/python3.8/site-packages/ray/_private/function_manager.py", line 480, in _wait_for_function
    self._worker.import_thread._do_importing()
AttributeError: 'Worker' object has no attribute 'import_thread'
An unexpected internal error occurred while the worker was executing a task.
(raylet) Traceback (most recent call last):
  File "python/ray/_raylet.pyx", line 2281, in ray._raylet.task_execution_handler
  File "python/ray/_raylet.pyx", line 2161, in ray._raylet.execute_task_with_cancellation_handler
  File "/home/josh/work/.../venv/lib/python3.8/site-packages/ray/_private/function_manager.py", line 391, in get_execution_info
    self._wait_for_function(function_descriptor, job_id)
  File "/home/josh/work/.../venv/lib/python3.8/site-packages/ray/_private/function_manager.py", line 480, in _wait_for_function
    self._worker.import_thread._do_importing()
AttributeError: 'Worker' object has no attribute 'import_thread'
An unexpected internal error occurred while the worker was executing a task. [repeated 18x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)

[jfaust]

Answered my own question - that fix did not make it into the 2.10 release