[core] (cgroups 8/n) Wiring CgroupManager into the raylet. #56297

israbbani · 2025-09-05T22:55:55Z

This PR stacks on #56285.

For more details about the resource isolation project see #54703.

This PR integrates the CgroupManager with the raylet startup in main.cc. It includes

A cross-platform cgroup_manager bazel target that selectively compiles dependencies for linux/non-linux platforms. I prefer this design because it keeps targets small and compile times low and removes unnecessary ifdefs from the code.
Adds the following command-line args to the raylet binary
- enable_resource_isolation
- cgroup_path
- system_reserved_cpu_weight
- system_reserved_memory_bytes

I've left comments on the PR to highlight parts that could use some feedback/discussion.

and core worker Signed-off-by: irabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

…irabbani/cgroups-7

a noop build target for non-linux platforms. Signed-off-by: irabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

israbbani · 2025-09-06T18:07:59Z

src/ray/raylet/main.cc

+#ifndef __linux__
+    RAY_LOG(WARNING)
+        << "Resource isolation with cgroups is only supported in linux. Please set "
+           "enable_resource_isolation to false. This is likely a misconfiguration.";
+#endif


I thought it would be useful to add a warning when resource isolation is enabled on non-linux platforms. It's likely a misconfiguration.

Signed-off-by: Ibrahim Rabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

Signed-off-by: Ibrahim Rabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

…irabbani/cgroups-8

ZacAttack · 2025-09-11T16:49:44Z

src/ray/raylet/main.cc

 DEFINE_string(labels,
              "",
              "Define the key-value format of node labels, which is a serialized JSON.");
+DEFINE_bool(


Thinking out loud, do we need this boolean flag? Could we infer it to be true if cgroup_path and other configs are present?

Fair point. I see the argument for both. More flags means more configuration possibilities. In this case, I picked an enable_ flag because the config is complex and it simplifies invariant checking and understanding the user's intent.

Ok. I'm not particularly opinionated one way or the other, mostly I guess probing to see if there was a reason to this rhyme and it sounds like you had one in mind. So we can keep this for now on the notion that it'll make it easier to extend later should that come along.

ZacAttack · 2025-09-11T16:52:07Z

src/ray/raylet/main.cc

+        << cgroup_manager.ToString();
+
+#ifndef __linux__
+    RAY_LOG(WARNING)


Do we need to do a system check to see if cgroups are enabled on the host?

This warning is for if the user is trying to enable resource isolation on a non-linux system. So Cgroups will not be enabled.

The line above this will create the appropriate target of CgroupManager based on the platform. If it's linux, it will fail if cgroups are not enabled on the host.

RAY_CHECK(cgroup_manager.ok()) << "Failed to start raylet. Could not create CgroupManager because of " << cgroup_manager.ToString();

I misspoke. The following is implemented in 9/n:

The line above this will create the appropriate target of CgroupManager based on the platform.

In this PR, we still do a system check if cgroups are enabled on the host in CgroupManager::Create.

…56297)" This reverts commit 161dd95.

#56352) This PR stacks on #56297. For more details about the resource isolation project see #54703. This PR wires the resource isolation config (introduced here #51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]>

ray-project#56352) This PR stacks on ray-project#56297. For more details about the resource isolation project see ray-project#54703. This PR wires the resource isolation config (introduced here ray-project#51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Zhiqiang Ma <[email protected]>

ray-project#56352) This PR stacks on ray-project#56297. For more details about the resource isolation project see ray-project#54703. This PR wires the resource isolation config (introduced here ray-project#51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: zac <[email protected]>

#56352) This PR stacks on #56297. For more details about the resource isolation project see #54703. This PR wires the resource isolation config (introduced here #51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: elliot-barn <[email protected]>

ray-project#56352) This PR stacks on ray-project#56297. For more details about the resource isolation project see ray-project#54703. This PR wires the resource isolation config (introduced here ray-project#51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Marco Stephan <[email protected]>

#56352) This PR stacks on #56297. For more details about the resource isolation project see #54703. This PR wires the resource isolation config (introduced here #51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: elliot-barn <[email protected]>

ray-project#56352) This PR stacks on ray-project#56297. For more details about the resource isolation project see ray-project#54703. This PR wires the resource isolation config (introduced here ray-project#51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Douglas Strodtman <[email protected]>

ray-project#56352) This PR stacks on ray-project#56297. For more details about the resource isolation project see ray-project#54703. This PR wires the resource isolation config (introduced here ray-project#51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]>

Original PR #56297 by israbbani Original: ray-project/ray#56297

… the raylet. Merged from original PR #56297 Original: ray-project/ray#56297

ray-project#56352) This PR stacks on ray-project#56297. For more details about the resource isolation project see ray-project#54703. This PR wires the resource isolation config (introduced here ray-project#51865) from ray cli (ray start) and the ray sdk (ray.init) into the raylet. Notable changes include: 1. A separate python test target for running test_resource_isolation related unit and integration tests. This unifies all cgroup related tests under one buildkite target and removes the need for `--except-tags cgroup` everywhere else. 2. Modification to the cgroup hierarchy. This was an oversight on my part. The "no internal processes" constraint says that a non-root cgroup can either have controllers enabled or have processes. Therefore, the new hierarchy looks like: ``` // base_cgroup_path (e.g. /sys/fs/cgroup) // | // ray_node_<node_id> // | | // system application // | | // leaf leaf // ``` where the leaf nodes contain all processes and the system/application cgroups apply cpu.weight and memory.min constraints. 3. CgroupManager now has a move ctor/assignment operator that allows ownership and lifecycle to be managed by the NodeManager. 4. CgroupManager is now owned by NodeManager. 5. An end-to-end integration test in `python/ray/tests/resource_isolation/test_resource_isolation_integration.py`. 6. Moved all previous integration tests from test_ray_init and test_cli into test_resource_isolation_integration.py. These tests have TODOs to finish them up once the rest of cgroup features are implemented. --------- Signed-off-by: irabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]>

israbbani added 5 commits September 5, 2025 16:22

[core] (cgroups 7/n) cleaning up old cgroup integration code for raylet

6fc9652

and core worker Signed-off-by: irabbani <[email protected]>

Merge branch 'irabbani/cgroups-6' into irabbani/cgroups-7

70ed06d

removing a few more references

fd39d7e

Signed-off-by: irabbani <[email protected]>

Merge branch 'irabbani/cgroups-7' of github.com:ray-project/ray into …

4e8e20c

…irabbani/cgroups-7

[core] (cgroups 8/n) Wiring CgroupManager into the raylet. Creating

9304014

a noop build target for non-linux platforms. Signed-off-by: irabbani <[email protected]>

israbbani added core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests labels Sep 5, 2025

israbbani mentioned this pull request Sep 5, 2025

[Core] Resource Isolation Milestone 1 Implementation Tracker #54703

Open

16 tasks

israbbani changed the title ~~[cores (cgroups 8/n) Wiring CgroupManager into the raylet.~~ [core] (cgroups 8/n) Wiring CgroupManager into the raylet. Sep 5, 2025

israbbani added 4 commits September 6, 2025 11:00

Merge branch 'irabbani/cgroups-6' into irabbani/cgroups-7

20bf0dd

Adding better error messages for creating the CgroupManager.

699f3ba

Signed-off-by: irabbani <[email protected]>

Removing node_manager configs

58e0c1a

Signed-off-by: irabbani <[email protected]>

unnecessary comment

e2e957d

Signed-off-by: irabbani <[email protected]>

israbbani commented Sep 6, 2025

View reviewed changes

Merge branch 'irabbani/cgroups-7' into irabbani/cgroups-8

9e7a2ef

israbbani marked this pull request as ready for review September 6, 2025 18:08

israbbani requested a review from a team as a code owner September 6, 2025 18:08

israbbani added 4 commits September 7, 2025 19:52

Merge branch 'irabbani/cgroups-6' into irabbani/cgroups-7

f86a010

Merge branch 'irabbani/cgroups-7' into irabbani/cgroups-8

29a7ae4

Merge branch 'irabbani/cgroups-6' into irabbani/cgroups-7

a843dd4

Signed-off-by: Ibrahim Rabbani <[email protected]>

Merge branch 'irabbani/cgroups-7' into irabbani/cgroups-8

43a180e

Signed-off-by: Ibrahim Rabbani <[email protected]>

israbbani mentioned this pull request Sep 8, 2025

[core] (cgroups 9/n) end-to-end integration of cgroups with ray start. #56352

Merged

bad merge

a3164a4

Signed-off-by: irabbani <[email protected]>

Base automatically changed from irabbani/cgroups-7 to irabbani/cgroups-6 September 9, 2025 13:54

israbbani added 4 commits September 9, 2025 10:34

Merge branch 'irabbani/cgroups-6' into irabbani/cgroups-8

9342ed5

Signed-off-by: Ibrahim Rabbani <[email protected]>

fixing ci

5b89821

Signed-off-by: irabbani <[email protected]>

Merge branch 'irabbani/cgroups-8' of github.com:ray-project/ray into …

4af9d3f

…irabbani/cgroups-8

Merge branch 'irabbani/cgroups-6' into irabbani/cgroups-8

74612b0

ZacAttack reviewed Sep 11, 2025

View reviewed changes

ZacAttack merged commit 161dd95 into irabbani/cgroups-6 Sep 11, 2025
4 checks passed

ZacAttack deleted the irabbani/cgroups-8 branch September 11, 2025 18:06

israbbani added a commit that referenced this pull request Sep 11, 2025

Revert "[core] (cgroups 8/n) Wiring CgroupManager into the raylet. (#…

0c87ca9

…56297)" This reverts commit 161dd95.

israbbani restored the irabbani/cgroups-8 branch September 11, 2025 18:22

israbbani deleted the irabbani/cgroups-8 branch September 11, 2025 18:47

snorkelopsstgtesting1-spec mentioned this pull request Oct 22, 2025

[core] (cgroups 8/n) Wiring CgroupManager into the raylet. snorkel-marlin-repos/ray-project_ray_pr_56297_6cb72018-61df-4d07-9fbe-47cf82d32e46#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[core] (cgroups 8/n) Wiring CgroupManager into the raylet. #56297

[core] (cgroups 8/n) Wiring CgroupManager into the raylet. #56297

Uh oh!

israbbani commented Sep 5, 2025 •

edited

Loading

Uh oh!

israbbani Sep 6, 2025

Uh oh!

ZacAttack Sep 11, 2025

Uh oh!

israbbani Sep 11, 2025

Uh oh!

ZacAttack Sep 11, 2025

Uh oh!

ZacAttack Sep 11, 2025

Uh oh!

israbbani Sep 11, 2025 •

edited

Loading

Uh oh!

israbbani Sep 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[core] (cgroups 8/n) Wiring CgroupManager into the raylet. #56297

[core] (cgroups 8/n) Wiring CgroupManager into the raylet. #56297

Uh oh!

Conversation

israbbani commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

israbbani Sep 6, 2025

Choose a reason for hiding this comment

Uh oh!

ZacAttack Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

israbbani Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

ZacAttack Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

ZacAttack Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

israbbani Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

israbbani Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

israbbani commented Sep 5, 2025 •

edited

Loading

israbbani Sep 11, 2025 •

edited

Loading