resource leak in ray/pthon/ray/node.py #9546

09wakharet · 2020-07-17T17:20:05Z

What is the problem?

The script below returns ResourceWarning: Enable tracemalloc to get the object allocation traceback many times. It seems to be a resource leak in ray/pthon/ray/node.py, due to unclosed logging files. The same problem can also be seen by running ray/rllib/utils/exploration/tests/test_explorations.py as-is.

Reproduction (REQUIRED)

import ray
import sys
import unittest

class TestResourceLeak(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        ray.init(local_mode=True)

    @classmethod
    def tearDownClass(cls):
        ray.shutdown()

    def test_useless(self):
        print("RAN")

if __name__ == "__main__":
    import pytest
    sys.exit(pytest.main(["-v", __file__]))

This will give the stack trace:

WARNING:tensorflow:From /opt/miniconda3/envs/ray/lib/python3.7/site-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2020-07-17 13:11:41,465	INFO resource_spec.py:223 -- Starting Ray with 4.44 GiB memory available for workers and up to 2.23 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis-shard_0.err' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis-shard_0.out' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis.err' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:738: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/redis.out' mode='a' encoding='utf-8'>
  self.start_redis()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:740: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/gcs_server.out' mode='a' encoding='utf-8'>
  self.start_gcs_server()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:740: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/gcs_server.err' mode='a' encoding='utf-8'>
  self.start_gcs_server()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:742: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/monitor.out' mode='a' encoding='utf-8'>
  self.start_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:742: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/monitor.err' mode='a' encoding='utf-8'>
  self.start_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2020-07-17 13:11:41,918	INFO services.py:1193 -- View the Ray dashboard at localhost:8265
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:747: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/dashboard.out' mode='a' encoding='utf-8'>
  self.start_dashboard(require_dashboard=False)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:747: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/dashboard.err' mode='a' encoding='utf-8'>
  self.start_dashboard(require_dashboard=False)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:755: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/plasma_store.out' mode='a' encoding='utf-8'>
  self.start_plasma_store()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:755: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/plasma_store.err' mode='a' encoding='utf-8'>
  self.start_plasma_store()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:756: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/raylet.out' mode='a' encoding='utf-8'>
  self.start_raylet()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:756: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/raylet.err' mode='a' encoding='utf-8'>
  self.start_raylet()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:757: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/reporter.out' mode='a' encoding='utf-8'>
  self.start_reporter()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:757: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/reporter.err' mode='a' encoding='utf-8'>
  self.start_reporter()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:760: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/log_monitor.out' mode='a' encoding='utf-8'>
  self.start_log_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:760: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-07-17_13-11-41_452687_79475/logs/log_monitor.err' mode='a' encoding='utf-8'>
  self.start_log_monitor()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 79521 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/subprocess.py:883: ResourceWarning: subprocess 79524 is still running
  ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/opt/miniconda3/envs/ray/lib/python3.7/site-packages/ray/node.py:978: ResourceWarning: unclosed file <_io.BufferedWriter name=8>
  allow_graceful=allow_graceful)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

If we cannot run your script, we cannot fix your issue.

I have verified my script runs in a clean environment and reproduces the issue.
I have verified the issue also occurs with the latest wheels.

The text was updated successfully, but these errors were encountered:

atomr3 · 2020-07-28T22:00:50Z

Great find

chunyang-wen · 2020-09-04T02:30:51Z

Same problem met. Just call ray.init() in one of the test functions, will get this warning.

miguel-angel-monjas · 2020-09-17T14:56:33Z

Same comment with pretty similar code:

class RayTestMethods(unittest.TestCase):

    def setUp(self):
        self.ray_client = RayClient()

    def test_model_addition(self):
        pass

    def tearDown(self):
        self.ray_client.shutdown()

rkooo567 · 2020-11-14T20:34:37Z

cc @edoakes Wasn't this intentional? What's the best way to remove warning here?

rkooo567 · 2022-12-08T07:11:39Z

Couldn't reproduce the issue in the master (see output below). Closing it! Please reopen if you still see the issue.

Test session starts (platform: darwin, Python 3.8.13, pytest 5.4.3, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/sangbincho/work/ray
plugins: asyncio-0.16.0, anyio-3.6.1, sugar-0.9.4, timeout-2.1.0, shutil-1.7.0, rerunfailures-10.2, virtualenv-1.7.0, repeat-0.9.1, lazy-fixture-0.6.3
collecting ... 
 a.py::TestResourceLeak.test_useless ✓                                                                                               100% ██████████
================================================================= warnings summary =================================================================
a.py::TestResourceLeak::test_useless
  /Users/sangbincho/work/ray/python/ray/_private/client_mode_hook.py:105: DeprecationWarning: DeprecationWarning: local mode is an experimental feature that is no longer maintained and will be removed in the future.For debugging consider using Ray debugger. 
    return func(*args, **kwargs)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

Results (1.50s):
       1 passed
(core) sangbincho@SangBins-MacBook-Pro-16-inch-2019 ray %

zhoubin-me · 2023-06-28T16:53:29Z

use pytest instead:

pytest pytest/test_unittest_xxx.py

dustinrb · 2024-01-04T20:53:36Z

Unfortunately this is still an issue.

Running a test job with unittest (python -m unittest test_ray.py) produces multiple ResourceWarnings like the original comment by @09wakharet.

# contents of test_ray.py
import ray
import unittest

class RayTest(unittest.TestCase):
    def setUp(self):
        self.ray_ctx = ray.init(num_cpus=1, num_gpus=0)

    def tearDown(self):
        ray.shutdown()

    def test_method(self):
        pass

I have tested this with Python 3.10 on macOS Big Sur/Ray==2.8 and macOS Sonoma/Ray==2.9.

samarth-contextual · 2024-02-18T21:53:44Z

Doesn't look like this got resolved.

python 3.10 // ubunutu 22.04 // CUDA 12.2

HenryL27 · 2024-05-14T22:53:37Z

I ran into this when integrating with the weaviate client:

import weaviate
import ray
ray.init()

python 3.11 // macos 14.4.1

hsm207 · 2024-05-15T00:10:04Z

I also ran into this when integration with the weaviate client.

Python 3.11.8 // Debian GNU/Linux 12 (bookworm) // ray 2.10.0

hsm207 · 2024-05-15T08:59:17Z

update:

I don't think this is actually related to the weaviate client.

I wrote a script named bug.py that has just this:

import ray
ray.init()

and ran it with:

python -Wd bug.py

and the warnings appear too.

Could the problem be that the log file streams created in open_log are not closed elsewhere?

09wakharet added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jul 17, 2020

richardliaw added P2 Important issue, but not time-critical and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Jul 17, 2020

rkooo567 self-assigned this Nov 13, 2020

JenishPatel99 mentioned this issue Jun 30, 2021

Warnings on starting evaluation process in ULTRA huawei-noah/SMARTS#980

Closed

amogkam mentioned this issue Jan 14, 2022

Ray scheduler driver and job api pytorch/torchx#329

Closed

jonathan-conder-sm mentioned this issue Feb 16, 2022

Fix ResourceWarnings #22419

Closed

6 tasks

rkooo567 closed this as completed Dec 8, 2022

rkooo567 reopened this Jan 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

resource leak in ray/pthon/ray/node.py #9546

resource leak in ray/pthon/ray/node.py #9546

09wakharet commented Jul 17, 2020

atomr3 commented Jul 28, 2020

chunyang-wen commented Sep 4, 2020

miguel-angel-monjas commented Sep 17, 2020

rkooo567 commented Nov 14, 2020

rkooo567 commented Dec 8, 2022

zhoubin-me commented Jun 28, 2023 •

edited

Loading

dustinrb commented Jan 4, 2024

samarth-contextual commented Feb 18, 2024

HenryL27 commented May 14, 2024 •

edited

Loading

hsm207 commented May 15, 2024

hsm207 commented May 15, 2024

resource leak in ray/pthon/ray/node.py #9546

resource leak in ray/pthon/ray/node.py #9546

Comments

09wakharet commented Jul 17, 2020

What is the problem?

Reproduction (REQUIRED)

atomr3 commented Jul 28, 2020

chunyang-wen commented Sep 4, 2020

miguel-angel-monjas commented Sep 17, 2020

rkooo567 commented Nov 14, 2020

rkooo567 commented Dec 8, 2022

zhoubin-me commented Jun 28, 2023 • edited Loading

dustinrb commented Jan 4, 2024

samarth-contextual commented Feb 18, 2024

HenryL27 commented May 14, 2024 • edited Loading

hsm207 commented May 15, 2024

hsm207 commented May 15, 2024

zhoubin-me commented Jun 28, 2023 •

edited

Loading

HenryL27 commented May 14, 2024 •

edited

Loading