[core] (cgroups 3/n) Creating CgroupManager to setup Ray's cgroup hierarchy and clean it up #56186

israbbani · 2025-09-03T01:37:44Z

This is the first PR introduces the CgroupManagerInterface. This will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs.

This PR stacks on #55063.

For more details about the resource isolation project see #54703.

The cgroup hierarchy for Ray will be:

    base_cgroup_path (e.g. /sys/fs/cgroup)
            |
    ray_node_<node_id>
      |           |
    system     application

The current implementation will only support

the cpu and memory controllers
cpu.weights and memory.min constraints.

I've signposted design decisions with comments in the code. Here's a summary:

CgroupManager::Create is a factory function. It is required because it can return a Status before creating an instance of CgroupManager if cgroupv2 is not setup correctly.
CgroupManager takes ownership of a CgroupDriverInterface for dependency injection.
CgroupManager uses ScopedCgroupOperations to register all side-effects to the cgroup hierarchy so graceful cleanup can apply them in reverse.

There are placeholders in the code for future work:

CgroupDriverInterface needs to implement a few more methods (DeleteCgroup, DisableController) to allow for full cleanup in CgroupManager.
CgroupManager needs additional tests for cleanup.
CgroupManagerInterface needs to expose APIs for moving processes into a cgroup.

to perform cgroup operations. Signed-off-by: irabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

instead of clone for older kernel headers < 5.7 (which is what we have in CI) Signed-off-by: irabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

…irabbani/cgroups-1

Signed-off-by: irabbani <[email protected]>

Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

…irabbani/cgroups-1

Signed-off-by: irabbani <[email protected]>

fix CI. Signed-off-by: irabbani <[email protected]>

Signed-off-by: irabbani <[email protected]>

Signed-off-by: Ibrahim Rabbani <[email protected]>

israbbani · 2025-09-04T20:30:31Z

Test failures look unrelated (from serve)

edoakes · 2025-09-04T21:08:48Z

src/ray/common/cgroup2/BUILD.bazel

+    hdrs = [
+        "cgroup_manager_interface.h",
+    ],
+    target_compatible_with = [


src/ray/common/cgroup2/cgroup_manager.cc

edoakes · 2025-09-04T21:11:46Z

src/ray/common/cgroup2/cgroup_manager.cc

+  int64_t max_cpu_weight = supported_constraints_.at(kCPUWeightConstraint).Max();
+  int64_t application_cgroup_cpu_weight = max_cpu_weight - system_reserved_cpu_weight;
+
+  RAY_LOG(INFO) << absl::StrFormat(


seems a little noisy for an info level, but only once per raylet startup so should be ok

As an SRE, I found log lines like this one very useful when debugging issues. As a rule of thumb I think we should log the configuration each component starts up with (especially if it's only created once in the lifecycle of the application).

Sounds good. This one is nice that all of the info is logged in one place. We have some other startup logs that are noisy because we log each bit in a separate log line from different components.

edoakes · 2025-09-04T21:12:21Z

src/ray/common/cgroup2/cgroup_manager.cc

+#include "ray/common/cgroup2/scoped_cgroup_operation.h"
+#include "ray/common/status_or.h"
+
+namespace ray {


maybe should have a namespace cgroup ?

I'm open to trying it. I haven't really wrapped my head around what best practices should be around namespaces. I've added an item to #54703. I'll play around with it at the end.

edoakes · 2025-09-04T21:12:42Z

src/ray/common/cgroup2/cgroup_manager.h

+      3. move all processes from the system cgroup into the base cgroup.
+      4. delete the node, system, and application cgroups respectively.
+
+    Cleanup is best-effort. If any step fails, it will log a warning.


why's this? to avoid crashing before other cleanup can happen at a higher level?

Yep. I figured we'd want to attempt the rest of the graceful shutdown process of the raylet even if cgroup cleanup didn't succeed fully.

edoakes · 2025-09-04T21:14:33Z

src/ray/common/cgroup2/fake_cgroup_driver.h

+           enabled_controllers_ == other.enabled_controllers_;
+  }
+};
+class FakeCgroupDriver : public CgroupDriverInterface {


edoakes · 2025-09-04T21:16:55Z

minor comments; ping for merge

Signed-off-by: irabbani <[email protected]>

israbbani · 2025-09-05T16:47:22Z

@edoakes ready for merge. Thanks!

…anager. (#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on #56186. For more details about the resource isolation project see #54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: sampan <[email protected]>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: sampan <[email protected]>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: jugalshah291 <[email protected]>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: jugalshah291 <[email protected]>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: yenhong.wong <[email protected]>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: yenhong.wong <[email protected]>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: zac <[email protected]>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: zac <[email protected]>

…rarchy and clean it up (#56186) Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Douglas Strodtman <[email protected]>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Douglas Strodtman <[email protected]>

…rarchy and clean it up (ray-project#56186) Signed-off-by: irabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]>

…anager. (ray-project#56246) This PR continues to implement the CgroupManager. CgroupManager will be used by the Raylet to manage the cgroup hierarchy. The implementation will be completed in subsequent PRs. This PR stacks on ray-project#56186. For more details about the resource isolation project see ray-project#54703. In this PR: * CgroupManager now bound checks constraints (e.g. cpu.weight is within [1,10000]. * CgroupDriver no longer bound checks constraints. --------- Signed-off-by: Ibrahim Rabbani <[email protected]> Co-authored-by: Edward Oakes <[email protected]>

israbbani and others added 30 commits July 24, 2025 20:39

[core] (cgroups 1/n) Adding a sys/fs filesystem driver

05c4dbc

to perform cgroup operations. Signed-off-by: irabbani <[email protected]>

adding the copyright

645f9a0

Signed-off-by: irabbani <[email protected]>

Adding a fallback for creating processes inside cgroups with fork/exec

2bb2c5b

instead of clone for older kernel headers < 5.7 (which is what we have in CI) Signed-off-by: irabbani <[email protected]>

adding a pause in the tests to see what's up with the container

4793094

Signed-off-by: irabbani <[email protected]>

Update src/ray/common/cgroup2/cgroup_driver_interface.h

85d0ebf

Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]>

Comments

3a5a020

Signed-off-by: irabbani <[email protected]>

Merge branch 'irabbani/cgroups-1' of github.com:ray-project/ray into …

68b0c93

…irabbani/cgroups-1

Putting the cgroupv2 tests into a separate target

f52354b

Signed-off-by: irabbani <[email protected]>

removing test sleep

148d04d

Signed-off-by: irabbani <[email protected]>

Removing a docstring

d3f8b79

Signed-off-by: irabbani <[email protected]>

enabling CI tests

d76ff15

Signed-off-by: irabbani <[email protected]>

fixing absl imports

2798ea5

Signed-off-by: irabbani <[email protected]>

commenting local

3fda505

Signed-off-by: irabbani <[email protected]>

doxygen formatting

9e1e931

Signed-off-by: irabbani <[email protected]>

Merge branch 'master' into irabbani/cgroups-1

f066f34

removing integration tests

e6b4926

Signed-off-by: irabbani <[email protected]>

final cleanup

f4e0cb2

Signed-off-by: irabbani <[email protected]>

iwyu

544ba83

Signed-off-by: irabbani <[email protected]>

Merge branch 'master' into irabbani/cgroups-1

669ba99

we cpplintin!

2e341d6

Signed-off-by: irabbani <[email protected]>

Update src/ray/common/cgroup2/sysfs_cgroup_driver.cc

9e46ce6

Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]>

Apply suggestions from code review

7c745c5

Co-authored-by: Edward Oakes <[email protected]> Signed-off-by: Ibrahim Rabbani <[email protected]>

bug

d7eb863

Signed-off-by: irabbani <[email protected]>

Merge branch 'irabbani/cgroups-1' of github.com:ray-project/ray into …

ff64534

…irabbani/cgroups-1

[core] Integration tests for SysFsCgroupDriver.

da4b475

Signed-off-by: irabbani <[email protected]>

Cleaning up cgroup_test_utils and attempting to

37e205f

fix CI. Signed-off-by: irabbani <[email protected]>

broken

7b83932

Signed-off-by: irabbani <[email protected]>

up

b911d25

Signed-off-by: irabbani <[email protected]>

upup

63506dc

Signed-off-by: irabbani <[email protected]>

Merge branch 'master' into irabbani/cgroups-2

e6f1ae9

Signed-off-by: Ibrahim Rabbani <[email protected]>

israbbani requested a review from a team as a code owner September 4, 2025 00:04

Merge branch 'irabbani/cgroups-2' into irabbani/cgroups-3

ca63baa

israbbani added the core Issues that should be addressed in Ray Core label Sep 4, 2025

Base automatically changed from irabbani/cgroups-2 to master September 4, 2025 04:54

israbbani added 2 commits September 4, 2025 09:28

Merge branch 'master' into irabbani/cgroups-3

0fe9113

Signed-off-by: Ibrahim Rabbani <[email protected]>

Merge branch 'master' into irabbani/cgroups-3

ca83426

israbbani mentioned this pull request Sep 4, 2025

[core] (cgroups 4/n) adding constraint bounds checking to the CgroupManager. #56246

Merged

edoakes reviewed Sep 4, 2025

View reviewed changes

src/ray/common/cgroup2/BUILD.bazel

hdrs = [

"cgroup_manager_interface.h",

],

target_compatible_with = [

Copy link

Collaborator

edoakes Sep 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🫶

edoakes approved these changes Sep 4, 2025

View reviewed changes

israbbani and others added 3 commits September 4, 2025 15:09

Merge branch 'master' into irabbani/cgroups-3

2457558

Addressing feedback.

b5f6c5e

Signed-off-by: irabbani <[email protected]>

Merge branch 'master' into irabbani/cgroups-3

03c731e

jjyao merged commit 5e85227 into master Sep 5, 2025
5 checks passed

jjyao deleted the irabbani/cgroups-3 branch September 5, 2025 17:57

[core] (cgroups 3/n) Creating CgroupManager to setup Ray's cgroup hierarchy and clean it up #56186

[core] (cgroups 3/n) Creating CgroupManager to setup Ray's cgroup hierarchy and clean it up #56186

Uh oh!

Conversation

israbbani commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

israbbani commented Sep 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edoakes commented Sep 4, 2025

Uh oh!

israbbani commented Sep 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

israbbani commented Sep 3, 2025 •

edited

Loading