[Feature] Support to run Ray worker container in a rootless container

[chenk008]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Now we can start Ray process in an isolated container, but root is required. It is unfortunate.

We need to run Ray worker rootless container. In Ray worker scenarios, what we need is an isolated rootfs and some cgroup configs.

rootfs

We can reuse container image, and use overlay to create a new rootfs.

In Ubuntu Distro (linux kernel 5.0+), we can mount a overlayfs in unprivileged user-ns. But other distro like centos/fedora not support it now.

cgroup

For cgroupv2, a unprivileged can create his sub cgroup.###

Use case

In a multi-tenant Ray cluster, each job may require different libraries and runtime. And immutable infrastructure can help users to deploy their job easily.

The container image provide an isolated filesystem and it is immutable. In most scenarios, Ray run without privilege, so it can't create normal container like Docker.

We need to support run worker in a slim container.

Related issues

#14077

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[chenk008]

Fuse-overlayfs is an implementation of overlay+shiftfs in FUSE for rootless containers, it can help to workaround this rootfs issue on non-debian based distros.

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!

[peterghaddad]

Can we re-open this issue?