Skip to content

Add unified /api/healthz#56943

Merged
edoakes merged 14 commits intoray-project:masterfrom
spencer-p:unified-health-pr
Dec 4, 2025
Merged

Add unified /api/healthz#56943
edoakes merged 14 commits intoray-project:masterfrom
spencer-p:unified-health-pr

Conversation

@spencer-p
Copy link
Contributor

@spencer-p spencer-p commented Sep 25, 2025

Why are these changes needed?

This new endpoint on the HealthzAgent class combines the status of /api/local_raylet_healthz and /api/gcs_healthz into one endpoint for use with Kubernetes.

Related issue number

See #56204.

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@spencer-p spencer-p requested a review from a team as a code owner September 25, 2025 23:48
@spencer-p spencer-p marked this pull request as draft September 25, 2025 23:49
gemini-code-assist[bot]
gemini-code-assist bot reviewed Sep 25, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new unified health check endpoint /api/healthz which combines the health status of the local Raylet and the GCS. The implementation is clear and achieves its goal. I have a couple of suggestions to improve code quality: one regarding Python's comparison operators for better correctness and another to remove some unreachable code for better clarity and maintainability.

edoakes
edoakes reviewed Sep 27, 2025
View reviewed changes
edoakes
edoakes reviewed Sep 27, 2025
View reviewed changes
@spencer-p spencer-p force-pushed the unified-health-pr branch 2 times, most recently from 69de984 to 5ace9dd Compare October 1, 2025 18:47
@edoakes edoakes added the go add ONLY when ready to merge, run all tests label Oct 1, 2025
edoakes
edoakes reviewed Oct 1, 2025
View reviewed changes
Copy link
Collaborator

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add some basic tests. If it's possible to write integration tests for this, that would be great, but not sure how because if the raylet and/or GCS become unhealthy, I think the agent will shortly crash...

Else we can scaffold up some unit tests (without too much mocking please)

@edoakes
Copy link
Collaborator

edoakes commented Oct 1, 2025

I also kicked off premerge CI by adding the go label, it'll run on each commit now: https://buildkite.com/ray-project/premerge/builds/50330

@github-actions
Copy link

github-actions bot commented Oct 16, 2025

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Oct 16, 2025
@edoakes
Copy link
Collaborator

edoakes commented Oct 16, 2025

unstale

@github-actions github-actions bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Oct 17, 2025
@400Ping
Copy link
Contributor

400Ping commented Nov 19, 2025

Hi any updates on this?

@spencer-p
Add unified /api/healthz
c85d4c6 
This new endpoint on the HealthzAgent class combines the status of
/api/local_raylet_healthz and /api/gcs_healthz into one endpoint for use
with Kubernetes. See ray-project#56204.

Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
fix 'is not' -> != for int
01c5b0a 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Add DashboardAgent is_head property
609bd5e 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Check health concurrently, no remote GCS check
22fb0fd 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Refactor unified health check; add logging
a016168 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
py format
a682247 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Fix unified health check
21b4e61 
- imports
- Use asyncio.gather instead of TaskGroup, which is not available on
  3.10
- Fix missing happy path in GCS check

Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Dashboard agent fetches is_head from GCS
20c26eb 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Track health check failures w/ exceptions
9abfe0a 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
lint
8b28cd7 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p
Unified health check tests
fe75776 
fixes missing await and bad logging references.

Signed-off-by: Spencer Peterson <spencerjp@google.com>
@spencer-p spencer-p marked this pull request as ready for review November 25, 2025 01:55
@spencer-p
Copy link
Contributor Author

spencer-p commented Nov 25, 2025

Tests are up; ready for review.

cursor[bot]
cursor bot reviewed Nov 25, 2025
View reviewed changes
@spencer-p spencer-p changed the title [WIP] Add unified /api/healthz Add unified /api/healthz Nov 25, 2025
@ray-gardener ray-gardener bot added core Issues that should be addressed in Ray Core observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling community-contribution Contributed by the community labels Nov 25, 2025
@jjyao jjyao requested a review from edoakes November 25, 2025 23:08
@spencer-p
dashboard asserts its own nodeinfo exists
c556707 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
cursor[bot]
cursor bot reviewed Nov 25, 2025
View reviewed changes
@spencer-p
retry node info fetch for 30s at most
9912df4 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
400Ping
400Ping approved these changes Nov 26, 2025
View reviewed changes
Copy link
Contributor

@400Ping 400Ping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice.

andrewsykim
andrewsykim reviewed Nov 26, 2025
View reviewed changes
@spencer-p
switch for loop for call_with_retry
563ff8a 
Signed-off-by: Spencer Peterson <spencerjp@google.com>
@edoakes edoakes merged commit 888083b into ray-project:master Dec 4, 2025
6 checks passed
@spencer-p spencer-p mentioned this pull request Dec 5, 2025
Open
4 tasks
@spencer-p spencer-p mentioned this pull request Jan 28, 2026
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewsykim andrewsykim andrewsykim left review comments

@cursor cursor[bot] cursor[bot] left review comments

@edoakes edoakes edoakes approved these changes

+2 more reviewers

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

@400Ping 400Ping 400Ping approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@edoakes edoakes

Labels

community-contribution Contributed by the community core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@spencer-p @edoakes @400Ping @andrewsykim