Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exceptions and ResourceWarnings on ray.init (Jupyter+offline) #10279

Open
austinmw opened this issue Aug 24, 2020 · 2 comments
Open

Exceptions and ResourceWarnings on ray.init (Jupyter+offline) #10279

austinmw opened this issue Aug 24, 2020 · 2 comments
Labels
bug Something that is supposed to be working; but isn't P3 Issue moderate in impact or severity

Comments

@austinmw
Copy link

austinmw commented Aug 24, 2020 •
edited
Loading

What is the problem?

I'm running ray in a JupyterLab notebook (AWS SageMaker), and the notebook has no internet connection (for data safety).
(instance type is ml.p3.16xlarge, so 64 vCPU and 488GiB ram).

When I simply do:

import ray
ray.init()

I get the following output:

2020-08-24 01:29:24,153	INFO resource_spec.py:212 -- Starting Ray with 323.83 GiB memory available for workers and up to 142.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis-shard_0.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/redis.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet_monitor.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/monitor.err' mode='a' encoding='UTF-8'>
2020-08-24 01:29:24,500	INFO services.py:1148 -- View the Ray dashboard at localhost:8265
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/dashboard.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/plasma_store.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/raylet.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/reporter.err' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.out' mode='a' encoding='UTF-8'>
Exception ignored in: <_io.FileIO name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='ab' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-08-24_01-29-24_152122_31504/logs/log_monitor.err' mode='a' encoding='UTF-8'>
{'node_ip_address': '10.54.198.48',
 'redis_address': '10.54.198.48:45400',
 'object_store_address': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/plasma_store',
 'raylet_socket_name': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504/sockets/raylet',
 'webui_url': 'localhost:8265',
 'session_dir': '/tmp/ray/session_2020-08-24_01-29-24_152122_31504'}

Do I need to modify any init settings due to the fact that I'm running in Jupyter and/or because I'm offline?

Ray version and other system information (Python version, TensorFlow version, OS):

  • Python 3.6.10
  • ray 0.8.7
@austinmw austinmw added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Aug 24, 2020
@rkooo567
Copy link
Contributor

Is there any way what exceptions are ignored in your logs?

@rkooo567 rkooo567 added P3 Issue moderate in impact or severity and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Aug 28, 2020
@edoakes
Copy link
Contributor

edoakes commented Aug 28, 2020

@austinmw this is a warning that's printed when a Python file object is created and the file isn't closed before the object is destructed. In most versions of Python this won't be printed, but it will if you're using a debug build or pass -W (and maybe some other circumstances, haven't seen it though).

This is intentional behavior because we need these files to be long-lived to write logs to, so there's nothing to worry about. Unfortunately I can't think of a good way to silence this at the moment. Leaving this issue open for others that might come across it.

@jonathan-conder-sm jonathan-conder-sm mentioned this issue Feb 16, 2022
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something that is supposed to be working; but isn't P3 Issue moderate in impact or severity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@edoakes @austinmw @rkooo567