[RayCluster] Status includes head containter status message

[spencer-p]

Why are these changes needed?

Sometimes cluster provisioning fails at runtime and requires intervention, but
it is not apparent from reading the RayCluster status that things will not
progress.

For example, users may forget to add image pull secrets or make a typo in their
container images and would only see ContainersNotReady, which may leave them
waiting for readiness that won't occur.

This change adds container status messages from Waiting and Terminated states to
the head pod condition to aid debuggability in these cases.

Before:

status:
  conditions:
  - lastTransitionTime: '2025-10-24T21:09:52Z'
    message: 'containers with unready status: [ray-head]'
    reason: ContainersNotReady
    status: 'False'
    type: HeadPodReady

After:

status:
  conditions:
  - lastTransitionTime: '2025-10-24T21:09:52Z'
    message: 'containers with unready status: [ray-head]; ray-head:
        ImagePullBackOff: Back-off pulling image "rayproject/roy:2.46.0":
        ErrImagePull: rpc error: code = NotFound desc = ...'
    reason: ContainersNotReady
    status: 'False'
    type: HeadPodReady

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[andrewsykim]

@spencer-p any reason for not updating worker status too? There's a separate condition for all workers that we can use I think

[kevin85421]

We had some earlier discussions: #2387 (comment).

This is a reasonable request because it has been requested several times, but we should be careful and understand: (1) why the K8s ReplicaSet didn't surface the issues, (2) when we should surface the issues, and (3) what we should surface.

[kevin85421]

Closes #2387

[rueian]

Hi @spencer-p, could you help fix the lint error?

[spencer-p]

any reason for not updating worker status too?

It was not as obvious low hanging fruit :)

I'd like to get some agreement on the shape of the change before I go ahead and do the same for the workers, but I'll consider that a goal for this PR.

we should be careful and understand: (1) why the K8s ReplicaSet didn't surface the issues, (2) when we should surface the issues, and (3) what we should surface.

I agree, and I'm a little wary of inserting all this info in the message string where it is not even that useful for automation. What do you think about inserting the container status reason and the message in only cases where the pod readiness reason is "ContainersNotReady"? E.g.

status:
  conditions:
  - lastTransitionTime: '2025-10-24T21:09:52Z'
	message: 'containers with unready status: [ray-head]; ray-head: Back-off
		pulling image "rayproject/roy:2.46.0": ErrImagePull: rpc error: code =
		NotFound desc = ...'
    reason: ImagePullBackoff
    status: 'False'
    type: HeadPodReady

Maybe it's a subtle point but I am more comfortable with this. I feel it addresses point 2 in narrowing the scope and point 3 in providing structured info (as well as the human readable msg)

(I'm not sure what you're referring to regarding replicasets (i didn't think kuberay used repliasets and I don't see any references?))

[kevin85421]

What do you think about inserting the container status reason and the message in only cases where the pod readiness reason is "ContainersNotReady"

This is reasonable to me.

(I'm not sure what you're referring to regarding replicasets (i didn't think kuberay used repliasets and I don't see any references?))

Oh, when I am not sure whether some changes are good or not, I will check upstream K8s behavior. For example, ReplicaSet also includes a group of Pods (similar to RayCluster), but it doesn't surface the issues of Pods in ReplicaSet's status.

[rueian]

Are we going to extend the RayClusterReplicaFailure condition? I think it should be in another PR due to the complexity.

[spencer-p]

Considering this ready for code review. I added the same behavior as the head pod to the workers (per Andrew's comment), where the first container failure for ContainersNotReady is reported.

Showing only the first error should help clue in new users and will avoid cases where hundreds of workers have errors concatenated together.

check upstream K8s behavior

I agree. I noticed Deployments will simply report MinimumReplicasUnavailable even if the pods are crashing or unpullable. I think it's been expected that Kubernetes-native users will know to dig deeper, but that's tricky for folks that are new (or more interested in their Ray job than pod statuses).

Are we going to extend the RayClusterReplicaFailure condition? I think it should be in another PR due to the complexity.

Just saw this after adding the change :). I'm happy to move it to another PR but it seems to be about the same complexity as the head pod change IMO.

[rueian]

Hi @spencer-p, can we have a new condition name for workers? WorkerPodsReady? Since not being ready yet may not result in a failure.

[spencer-p]

I like WorkerPodsReady.

That opens up some new cases when there are no workers. I am thinking in such cases the condition should be omitted.

[spencer-p]

Reverting the last commit so that WorkerPodsReady can have a separate PR as it will be more detailed

[andrewsykim]

Just some nits

[andrewsykim]

nit: named returned variables are not needed here

[spencer-p]

I named the return variables for documentation, to disambiguate the two strings

[andrewsykim]

nit: return status.State.Waiting.Reason, fmt.Sprintf("%s: %s", status.Name, status.State.Waiting.Message), true

[andrewsykim]

(same for terminated)

[spencer-p]

Done