Add support for Ray token auth

[andrewsykim]

Why are these changes needed?

TODO:

• unit / integration tests
• support for referencing an existing secret

Related issue number

#4203

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[andrewsykim]

TODO after ray-project/ray#58408

[andrewsykim]

TODO after ray-project/ray#58408

[rueian]

Suggested change

	Key: "auth_token",
	Key: utils.RAY_AUTH_TOKEN_SECRET_KEY,

[andrewsykim]

fixed, thanks

[rueian]

Suggested change

	func IsAuthEnabled(spec *rayv1.RayClusterSpec) bool {
	func IsTokenAuthEnabled(spec *rayv1.RayClusterSpec) bool {

[andrewsykim]

done

[rueian]

Weird. I am still seeing it remains the old IsAuthEnabled.

[andrewsykim]

sorry, I probably forgot to push the latest commits

[rueian]

Suggested change

	func setAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {
	func setTokenAuthEnvVars(clusterName string, podTemplate *corev1.PodTemplateSpec) {

[rueian]

nit: a bit more explicit.

[andrewsykim]

done

[Future-Outlier]

Would it be possible to include an example (under folder ray-operator/config/samples/) using rayproject/ray:nightly to test this PR prior to merging?

[Future-Outlier]

I am thinking is ray_auth_token better than auth_token?

[Future-Outlier]

I use the following example to test this PR.
And I found that the RayCluster's head pod is never READY, do you know why?

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-kuberay-auth-token
spec:
  rayVersion: "2.51.0"
  authOptions:
    mode: "token"
  headGroupSpec:
    # rayStartParams is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:nightly
          resources:
            # limits:
            #   cpu: 1
            #   memory: 2G
            requests:
              cpu: 1
              memory: 2G
          ports:
          - containerPort: 6379
            name: gcs-server
          - containerPort: 8265 # Ray dashboard
            name: dashboard
          - containerPort: 10001
            name: client
  workerGroupSpecs:
  # the pod replicas in this group typed worker
  - replicas: 1
    minReplicas: 1
    maxReplicas: 5
    # logical group name, for this called small-group, also can be functional
    groupName: workergroup
    # rayStartParams is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc'
          image: rayproject/ray:nightly
          resources:
            # limits:
            #   cpu: 1
            #   memory: 1G
            requests:
              cpu: 1
              memory: 1G

[Future-Outlier]

Suggested change

	Value: "token",
	})
	Value: rayv1.AuthModeToken,
	})

[Future-Outlier]

I have 2 questions relate to RBAC.

1. Our controller only calls Get() and Create() on Secrets, never delete, patch, or update. Should we remove these unused verbs from the RBAC definition to follow the principle of least privilege?
2. Should we add Owns(&corev1.Secret{}) to SetupWithManager? Currently we have list and watch permissions in RBAC, but without Owns(), these permissions aren’t being used.

(but I'm not sure watch and list secret is a good practice or not)

[andrewsykim]

We'll need watch/list for informer cache, good call on delete, we probably don't need that as we can rely on garbage collection from owner ref

[Future-Outlier]

I might try to build a ray image with this PR and test it tmr.

ray-project/ray#58368

[Future-Outlier]

We'll need watch/list for informer cache, good call on delete, we probably don't need that as we can rely on garbage collection from owner ref

sorry I didn't write my comments correctly, I agreed we need watch/list for informer cache, and should we add Owns(&corev1.Secret{}) to SetupWithManager?

[andrewsykim]

@Future-Outlier FYI if you're testing, you need to use nightly build and there's a lot of on-going changes for this feature

[Future-Outlier]

@Future-Outlier FYI if you're testing, you need to use nightly build and there's a lot of on-going changes for this feature

thanks @andrewsykim
I will test more tmr and will probably provide a video or screenshot

[Future-Outlier]

just successfully built the image

cd ray-project/ray
docker build --progress=plain --build-arg BUILD_DATE="$(date +%Y-%m-%d:%H:%M:%S)" -t $RAY_IMAGE -f ./python/ray/tests/kuberay/Dockerfile .

[andrewsykim]

@sampan-s-nayak is it expected for "ray healh-check` to require auth token?

[win5923]

Is ray autoscaler also require auth token?

[andrewsykim]

Good catch, I'll add this logic shortly

[win5923]

I think it’s better to set the default to disabled, as token authentication requires Ray >= 2.51.0 and some users may still be on older pinned versions.

[andrewsykim]

It's worth mentioning the mode only defaults to token when authOptions != nil

[win5923]

Is ray autoscaler also require auth token?

[Future-Outlier]

my current thought:

before this PR get merged
I think we should

1. test token mode
2. test k8s mode
3. test case with 1 and 2 + autoscaler

given that build a ray image is extremely hard, I'm going to wait for the latest ray image release.

[andrewsykim]

@Future-Outlier let's keep this PR scoped to token mode only. The k8s auth mode will be undocumented feature for a couple Ray releases since it needs more testing, we can add KubeRay support once the public docs are ready (probably before KubeRay v1.6 though)

[Future-Outlier]

Here's a demo video using latest ray image "rayproject/ray:nightly-py311-cpu"
(2.52.0 not released yet)
the ray's commit is ray-project/ray@79d2a69

I'm really happy that it works!

2025-11-17.15-52-20.mp4

my yaml file

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-kuberay-auth-token
spec:
  rayVersion: "2.51.0"
  authOptions:
    mode: "token"
  headGroupSpec:
    # rayStartParams is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-head
          image: rayproject/ray:nightly-py311-cpu
          resources:
            # limits:
            #   cpu: 1
            #   memory: 2G
            requests:
              cpu: 1
              memory: 2G
          ports:
          - containerPort: 6379
            name: gcs-server
          - containerPort: 8265 # Ray dashboard
            name: dashboard
          - containerPort: 10001
            name: client
  workerGroupSpecs:
  # the pod replicas in this group typed worker
  - replicas: 1
    minReplicas: 1
    maxReplicas: 5
    # logical group name, for this called small-group, also can be functional
    groupName: workergroup
    # rayStartParams is optional with RayCluster CRD from KubeRay 1.4.0 or later but required in earlier versions.
    rayStartParams: {}
    template:
      spec:
        containers:
        - name: ray-worker # must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc'
          image: rayproject/ray:nightly-py311-cpu
          resources:
            # limits:
            #   cpu: 1
            #   memory: 1G
            requests:
              cpu: 2
              memory: 1G

[Future-Outlier]

after changing "RAY_auth_mode" to "RAY_AUTH_MODE"
we have these follow-up to do

1. make this work with autoscaler
2. enable users to configure their own secret
3. rbac
4. make sure this work with rayjob (need to pass auth token)
   a. with cluster selector
   b. without cluster selector
   i. k8s job mode
   ii. http mode
   iii. interactive mode
   iv. sidecar mode
5. rayservice
6. validation
7. maybe also need to support kuberay API server (can be done in kuberay 1.6)

[andrewsykim]

Thanks for reviewing but I need to make a few more changes in this PR before merging:

• RAY_auth_mode -> RAY_AUTH_MODE
• Update cluster autoscaler

I think the other changes @Future-Outlier mentioned for RayJob and RayService can be done in follow-up PRs

[Future-Outlier]

autoscaler works in my manual testing.