[core] (2/n) [Removing GCS Centralized Scheduling] Removing ClusterLeaseManager from GCS. #60008

israbbani · 2026-01-09T19:25:43Z

This PR stacks on #59979.

This is 2/N in a series of PRs to remove Centralized Actor Scheduling by the GCS (introduced in #15943). The feature is off by default and no longer in use or supported.

In this PR, I remove use of the ClusterLeaseManager (from the Raylet's scheduler) in the GCS.

Signed-off-by: irabbani <[email protected]>

gemini-code-assist

Code Review

This pull request is part of a larger effort to remove the dependency on ClusterLeaseManager from GCS components. The changes primarily involve removing ClusterLeaseManager from GcsActorScheduler, GcsResourceManager, and GcsServer, and updating their dependencies and logic accordingly. While most of the changes look correct and consistent with the refactoring goal, I've identified a few potential issues related to resource reporting for autoscaling, missing defensive checks, and logic for triggering global garbage collection. These might be regressions or temporary states in a work-in-progress PR, but are worth pointing out.

I am having trouble creating individual review comments. Click here to see my feedback.

src/ray/gcs/gcs_resource_manager.cc (179-185)

The removal of this block means that pending actor resource demands are no longer included in the resource usage report. This could prevent the autoscaler from scaling up correctly for pending actors. Was this intentional? If this functionality is still needed, the pending actor information could be retrieved from GcsActorManager.

src/ray/gcs/gcs_server.cc (396)

This RAY_CHECK is removed. While the check for cluster_lease_manager_ is no longer needed, it would be safer to retain the check for cluster_resource_scheduler_ to ensure it's initialized before use.

  RAY_CHECK(cluster_resource_scheduler_);

src/ray/gcs/gcs_server.cc (518)

This RAY_CHECK is removed. While the check for cluster_lease_manager_ is no longer needed, it would be safer to retain a check for gcs_resource_manager_.

  RAY_CHECK(gcs_resource_manager_);

src/ray/gcs/gcs_server.cc (987-990)

This check for pending tasks is removed, which means TryGlobalGC might attempt to trigger a global GC even when there are no pending tasks. This seems inefficient. The check should be updated to use the new way of tracking pending actors, for example by checking gcs_actor_manager_->GetPendingActorsCount().

  if (gcs_actor_manager_->GetPendingActorsCount() == 0) {
    task_pending_schedule_detected_ = 0;
    return;
  }

israbbani · 2026-01-09T20:08:31Z

Reviewing Gemini's comments

src/ray/gcs/gcs_resource_manager.cc (179-185)

This is fine. There's nothing to include if you're not using the GCS as a global scheduler.

src/ray/gcs/gcs_server.cc (396)

Fair enough. I reintroduced this for now. We have too many RAY_CHECKs with no indication of why they're important.

src/ray/gcs/gcs_server.cc (518)

Nope. I think the original check was wrong because we don't use cluster_resource_manager_ inside InitGCSActorManager at all. It was probably a typo in the original code.

src/ray/gcs/gcs_server.cc (987-990)

I'm not convinced. The original wasn't calling the GcsActorScheduler, but the ClusterLeaseManager directly. This means that the current code wasn't checking from the perspective of the GCS being the owner for Actors, but rather the scheduler.

…abbani/remove-centralized-actor-scheduling-2

Signed-off-by: irabbani <[email protected]>

cursor · 2026-01-09T22:06:22Z

src/ray/gcs/gcs_server.cc

 }

 void GcsServer::TryGlobalGC() {
-  if (cluster_lease_manager_->GetPendingQueueSize() == 0) {


Global GC counter never resets, causing unnecessary GC triggers

Medium Severity

The TryGlobalGC function previously checked if cluster_lease_manager_->GetPendingQueueSize() == 0 and reset task_pending_schedule_detected_ to 0 when there were no pending tasks. With the ClusterLeaseManager removed, this check and reset are gone. The counter now increments on every call (every ~10 seconds by default) and never resets. After the first two calls, task_pending_schedule_detected_++ > 0 will always be true, causing global GC to be triggered whenever the throttler allows, regardless of whether there are actually pending tasks. This results in unnecessary global GC broadcasts across the cluster. The comment "To avoid spurious triggers" is now stale since spurious triggers can occur.

🔬 Verification Test

Why verification test was not possible: This bug involves runtime behavior in a distributed system (Ray GCS server). Testing would require spinning up a Ray cluster and observing GC trigger patterns over time, which is not feasible in a unit test context. The bug can be verified by code inspection: the removed lines (958-961 in the original) contained the counter reset logic task_pending_schedule_detected_ = 0; return; that is no longer present, while the increment task_pending_schedule_detected_++ at line 972 continues to execute on every call without any reset mechanism.

This is real. I'm deleting GlobalGC from the GCS completely in #60019. Keeping the PRs separate to make them easy to review.

@edoakes on second thought. It's a bad idea to separate these because now GC will run more frequently until #60019 is merged. Can we merge both together? I can combine them too..

+1 on combining

Dead code is a maintenance burden. Removing unused mocks. This came up as I was working on removing the ClusterLeaseManager from the GCS in #60008. Signed-off-by: irabbani <[email protected]>

Sparks0219 · 2026-01-10T00:53:31Z

src/ray/gcs/BUILD.bazel

        "//src/ray/protobuf:gcs_service_cc_grpc",
        "//src/ray/pubsub:gcs_publisher",
        "//src/ray/pubsub:publisher",
+        # TODO(irabbani): Refactor a subset of scheduling into a shared targer


super nit: typo

Sparks0219 · 2026-01-10T00:54:53Z

src/ray/gcs/gcs_actor_manager.cc

 size_t GcsActorManager::GetPendingActorsCount() const {
-  return gcs_actor_scheduler_->GetPendingActorsCount() + pending_actors_.size();
+  return pending_actors_.size();
+  // return gcs_actor_scheduler_->GetPendingActorsCount() + pending_actors_.size();


remove comment

Sparks0219 · 2026-01-10T00:56:02Z

src/ray/gcs/gcs_resource_manager.h

 #include "ray/gcs/grpc_service_interfaces.h"
 #include "ray/ray_syncer/ray_syncer.h"
-#include "ray/raylet/scheduling/cluster_lease_manager.h"
+// #include "ray/raylet/scheduling/cluster_lease_manager.h"


Sparks0219

LGTM, thanks for doing this!

[core] (2/n) Removing GCS's dependency on ClusterLeaseManager.

8747dcb

Signed-off-by: irabbani <[email protected]>

gemini-code-assist bot reviewed Jan 9, 2026

View reviewed changes

israbbani closed this Jan 9, 2026

israbbani reopened this Jan 9, 2026

Merge branch 'irabbani/remove-centralized-actor-scheduling-1' into ir…

ddf21ec

…abbani/remove-centralized-actor-scheduling-2

israbbani changed the title ~~[core] (2/n) Removing GCS's dependency on ClusterLeaseManager.~~ [core] (2/n) [Removing GCS Scheduling] Removing ClusterLeaseManager from GCS. Jan 9, 2026

israbbani mentioned this pull request Jan 9, 2026

[core] Deleting a bunch of unused GCS mocks. #60015

Merged

Removing comments and fixing a breaking test

1af17bf

Signed-off-by: irabbani <[email protected]>

israbbani marked this pull request as ready for review January 9, 2026 21:57

israbbani requested a review from a team as a code owner January 9, 2026 21:57

israbbani added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Jan 9, 2026

israbbani changed the title ~~[core] (2/n) [Removing GCS Scheduling] Removing ClusterLeaseManager from GCS.~~ [core] (2/n) [Removing GCS Centralized Scheduling] Removing ClusterLeaseManager from GCS. Jan 9, 2026

cursor bot reviewed Jan 9, 2026

View reviewed changes

edoakes approved these changes Jan 9, 2026

View reviewed changes

israbbani mentioned this pull request Jan 9, 2026

[core] (3/n) [Removing GCS Centralized Scheduling] Removing GlobalGC from the GCS #60019

Open

Sparks0219 reviewed Jan 10, 2026

View reviewed changes

Sparks0219 approved these changes Jan 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[core] (2/n) [Removing GCS Centralized Scheduling] Removing ClusterLeaseManager from GCS. #60008

[core] (2/n) [Removing GCS Centralized Scheduling] Removing ClusterLeaseManager from GCS. #60008

israbbani commented Jan 9, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

israbbani commented Jan 9, 2026 •

edited

Loading

Uh oh!

cursor bot Jan 9, 2026

Uh oh!

israbbani Jan 9, 2026

Uh oh!

israbbani Jan 10, 2026

Uh oh!

Sparks0219 Jan 10, 2026

Uh oh!

Sparks0219 Jan 10, 2026

Uh oh!

Sparks0219 Jan 10, 2026

Uh oh!

Sparks0219 Jan 10, 2026

Uh oh!

Sparks0219 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[core] (2/n) [Removing GCS Centralized Scheduling] Removing ClusterLeaseManager from GCS. #60008

Are you sure you want to change the base?

[core] (2/n) [Removing GCS Centralized Scheduling] Removing ClusterLeaseManager from GCS. #60008

Conversation

israbbani commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

src/ray/gcs/gcs_resource_manager.cc (179-185)

src/ray/gcs/gcs_server.cc (396)

src/ray/gcs/gcs_server.cc (518)

src/ray/gcs/gcs_server.cc (987-990)

Uh oh!

israbbani commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor bot Jan 9, 2026

Choose a reason for hiding this comment

Global GC counter never resets, causing unnecessary GC triggers

Uh oh!

israbbani Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

israbbani Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Sparks0219 Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Sparks0219 Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Sparks0219 Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Sparks0219 Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Sparks0219 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

israbbani commented Jan 9, 2026 •

edited

Loading

israbbani commented Jan 9, 2026 •

edited

Loading