Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Observability] Ray Dashboard and Metrics aren't listing Driver by default #50097

Open
alexeykudinkin opened this issue Jan 28, 2025 · 0 comments
Labels
bug Something that is supposed to be working; but isn't core-observability observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling P1 Issue that should be fixed within a few weeks triage Needs triage (eg: priority, bug/not-bug, and owning component)

Comments

@alexeykudinkin
Copy link
Contributor

What happened + What you expected to happen

Recently, i've run into issue where my Driver was going was running out of heap memory, and there's surprisingly no direct way to observe it on any of Ray's observability surfaces

  • Dashboard doesn't list driver as the process running on the head node
  • Metrics don't list driver as standalone component (though it lists dashboard, agent, etc)

This creates a gap where driver isn't really covered by any of the existing surfaces.

Versions / Dependencies

2.41

Reproduction script

Make driver script allocate and retain memory indefinitely
Try to find the source of the problem from Ray Dashboard or Metrics

Issue Severity

None

@alexeykudinkin alexeykudinkin added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling core-observability P1 Issue that should be fixed within a few weeks labels Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't core-observability observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling P1 Issue that should be fixed within a few weeks triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Development

No branches or pull requests

1 participant