[Core] Update cluster scheduler to handle label selector hard node id constraint #56235

ryanaoleary · 2025-09-04T10:00:36Z

Why are these changes needed?

This PR updates the cluster scheduler to check for ray.io/node-id label selectors when calling GetBestSchedulableNode. If a feasible node is found with the desired ID it's returned, and otherwise the resource demand is marked infeasible and nil NodeID is returned. Then in the ClusterLeaseManager, we are able to check for node ID label constraints and return an unschedulable error when scheduling_node_id.IsNil().

This behavior matches exactly how NodeAffinitySchedulingPolicy handles infeasible nodes when soft=False. This change is necessary for #54940 which replaces usages of NodeAffinitySchedulingPolicy, since otherwise the behavior of an unsatisfiable ray.io/node-id label selector is to remain pending indefinitely.

Related issue number

#51564

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Ryan O'Leary <[email protected]>

ryanaoleary · 2025-09-09T00:59:27Z

cc: @MengjinYan

src/ray/common/scheduling/label_selector.h

src/ray/raylet/scheduling/cluster_lease_manager.cc

src/ray/raylet/scheduling/cluster_resource_scheduler.cc

Signed-off-by: Ryan O'Leary <[email protected]>

src/ray/raylet/scheduling/cluster_resource_scheduler.cc

Signed-off-by: Ryan O'Leary <[email protected]>

jjyao

LGTM

jjyao · 2025-09-15T18:07:17Z

src/ray/core_worker/lease_policy.cc

+  if (auto node_id_values = GetHardNodeAffinityValues(spec.GetLabelSelector())) {
+    for (const auto &node_id_hex : *node_id_values) {
+      if (auto addr = node_addr_factory_(NodeID::FromHex(node_id_hex))) {
+        return std::make_pair(addr.value(), false);


Something we can do smarter in the follow-up is that if we have a list of nodes here, we should pick the node with the most arguments.

Actually, I'm wondering if the logic should be in here where the core_worker finds the raylet to send the requests to or it should be in the the raylet logic where it finds the best node to send the task?

It's a good question. Currently it has to be here since raylet doesn't have location information of objects (owner does).

src/ray/raylet/scheduling/cluster_lease_manager.cc

src/ray/raylet/scheduling/tests/cluster_resource_scheduler_test.cc

…t.cc Co-authored-by: Mengjin Yan <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]>

Signed-off-by: Ryan O'Leary <[email protected]>

MengjinYan · 2025-09-15T21:14:46Z


[2025-09-15T20:14:44Z] In file included from src/ray/common/scheduling/label_selector.cc:15:
--
  | [2025-09-15T20:14:44Z] bazel-out/k8-opt/bin/src/ray/common/scheduling/_virtual_includes/label_selector/ray/common/scheduling/label_selector.h:24:10: fatal error: ray/common/constants.h: No such file or directory
  | [2025-09-15T20:14:44Z]    24 \| #include "ray/common/constants.h"
  | [2025-09-15T20:14:44Z]       \|          ^~~~~~~~~~~~~~~~~~~~~~~~
  | [2025-09-15T20:14:44Z] compilation terminated.

Seems to be some compilation error.

Signed-off-by: Ryan O'Leary <[email protected]>

ryanaoleary · 2025-09-15T22:18:24Z

[2d29039](/ray-project/ray/pull/56235/commits/2d290393bac93ff41b001914ca70c7bba542ab36)

Looks like the BUILD.bazel file recently changed, 2d29039 should fix this by updating the deps.

jjyao · 2025-09-16T04:24:52Z


bazel-out/k8-opt/bin/src/ray/common/scheduling/_virtual_includes/label_selector/ray/common/scheduling/label_selector.h:24:10: fatal error: ray/common/constants.h: No such file or directory
--
  | [2025-09-16T00:18:42Z]    24 \| #include "ray/common/constants.h"
  | [2025-09-16T00:18:42Z]       \|          ^~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Ryan O'Leary <[email protected]>

ryanaoleary · 2025-09-16T10:31:23Z


bazel-out/k8-opt/bin/src/ray/common/scheduling/_virtual_includes/label_selector/ray/common/scheduling/label_selector.h:24:10: fatal error: ray/common/constants.h: No such file or directory
--
  | [2025-09-16T00:18:42Z]    24 \| #include "ray/common/constants.h"
  | [2025-09-16T00:18:42Z]       \|          ^~~~~~~~~~~~~~~~~~~~~~~~

This should be passing now.

MengjinYan · 2025-09-16T20:10:21Z

The java test failure should be unrelated. cc: @jjyao

… constraint (ray-project#56235) Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Jiajun Yao <[email protected]>

… constraint (ray-project#56235) Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: zac <[email protected]>

… constraint (ray-project#56235) Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: Marco Stephan <[email protected]>

… constraint (#56235) Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: Douglas Strodtman <[email protected]>

… constraint (ray-project#56235) Signed-off-by: Ryan O'Leary <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]> Co-authored-by: Mengjin Yan <[email protected]> Co-authored-by: Jiajun Yao <[email protected]>

ryanaoleary and others added 3 commits September 4, 2025 09:42

Update scheduler to handle label selector hard node id constraint

16bd8fd

Signed-off-by: Ryan O'Leary <[email protected]>

Merge branch 'master' into label-selector-node-id-constraint

d8226fc

Fix c++ test

9ce9890

Signed-off-by: Ryan O'Leary <[email protected]>

ryanaoleary marked this pull request as ready for review September 9, 2025 00:59

ryanaoleary requested a review from a team as a code owner September 9, 2025 00:59

ray-gardener bot added core Issues that should be addressed in Ray Core community-contribution Contributed by the community labels Sep 9, 2025

edoakes assigned MengjinYan Sep 9, 2025

MengjinYan reviewed Sep 9, 2025

View reviewed changes

src/ray/common/scheduling/label_selector.h Outdated Show resolved Hide resolved

src/ray/raylet/scheduling/cluster_lease_manager.cc Show resolved Hide resolved

src/ray/raylet/scheduling/cluster_resource_scheduler.cc Outdated Show resolved Hide resolved

Use constants value

0603f0e

Signed-off-by: Ryan O'Leary <[email protected]>

ryanaoleary requested a review from MengjinYan September 11, 2025 07:25

jjyao reviewed Sep 11, 2025

View reviewed changes

src/ray/raylet/scheduling/cluster_resource_scheduler.cc Outdated Show resolved Hide resolved

Remove early failure check

d088786

Signed-off-by: Ryan O'Leary <[email protected]>

jjyao added the go add ONLY when ready to merge, run all tests label Sep 13, 2025

jjyao reviewed Sep 15, 2025

View reviewed changes

MengjinYan approved these changes Sep 15, 2025

View reviewed changes

src/ray/raylet/scheduling/cluster_lease_manager.cc Show resolved Hide resolved

src/ray/raylet/scheduling/tests/cluster_resource_scheduler_test.cc Outdated Show resolved Hide resolved

ryanaoleary and others added 3 commits September 15, 2025 12:51

Update src/ray/raylet/scheduling/tests/cluster_resource_scheduler_tes…

f08c4c0

…t.cc Co-authored-by: Mengjin Yan <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]>

Clarify in comment that this is for infeasible resources

76a1f73

Signed-off-by: Ryan O'Leary <[email protected]>

Merge branch 'master' into label-selector-node-id-constraint

0877438

ryanaoleary and others added 2 commits September 15, 2025 22:16

Merge branch 'master' into label-selector-node-id-constraint

d741f50

Add common:constants dep

2d29039

Signed-off-by: Ryan O'Leary <[email protected]>

jjyao enabled auto-merge (squash) September 15, 2025 22:44

Merge branch 'master' into label-selector-node-id-constraint

c5bf081

github-actions bot disabled auto-merge September 15, 2025 23:45

ryanaoleary and others added 2 commits September 16, 2025 08:25

Fix imports

e6381e4

Signed-off-by: Ryan O'Leary <[email protected]>

Merge branch 'master' into label-selector-node-id-constraint

6b830ca

Merge branch 'master' into label-selector-node-id-constraint

4a9bd1f

jjyao enabled auto-merge (squash) September 16, 2025 16:46

Merge branch 'master' into label-selector-node-id-constraint

3e815a2

github-actions bot disabled auto-merge September 16, 2025 20:09

jjyao merged commit bfe8139 into ray-project:master Sep 16, 2025
3 of 5 checks passed

ryanaoleary mentioned this pull request Sep 16, 2025

[Core] Ray Label Selector API Implementation Tracker #51564

Open

35 tasks

[Core] Update cluster scheduler to handle label selector hard node id constraint #56235

[Core] Update cluster scheduler to handle label selector hard node id constraint #56235

Uh oh!

Conversation

ryanaoleary commented Sep 4, 2025

Why are these changes needed?

Related issue number

Checks

Uh oh!

ryanaoleary commented Sep 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjyao left a comment

Choose a reason for hiding this comment

Uh oh!

jjyao Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

MengjinYan Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

jjyao Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MengjinYan commented Sep 15, 2025

Uh oh!

ryanaoleary commented Sep 15, 2025

Uh oh!

jjyao commented Sep 16, 2025

Uh oh!

ryanaoleary commented Sep 16, 2025

Uh oh!

MengjinYan commented Sep 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants