[Feature] Support JobDeploymentStatus as the deletion condition #4262

JiangJiaWei1103 · 2025-12-07T23:26:15Z

Why are these changes needed?

The current deletionStrategy relies exclusively on the terminal states of JobStatus (SUCCEEDED or FAILED). However, there are several scenarios in which a user-deployed RayJob ends up with JobStatus == "" (JobStatusNew) while JobDeploymentStatus == "Failed". In these cases, the associated resources (e.g., RayJob, RayCluster, etc.) remain stuck and are never cleaned up, resulting in indefinite resource consumption.

Changes

Add the JobDeploymentStatus field to DeletionCondition
- Currently supports Failed only
Enforce mutual exclusivity between JobStatus and JobDeploymentStatus within DeletionCondition

Implementation Details

To determine which field the user specifies, we use pointers instead of raw values. Both JobStatus and JobDeploymentStatus have empty strings as their zero values, which correspond to a "new" state. Using nil allows us to reliably distinguish between "unspecified" and "explicitly set," avoiding unintended ambiguity.

Related issue number

Closes #4233.

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: JiangJiaWei1103 <[email protected]>

…cies Signed-off-by: JiangJiaWei1103 <[email protected]>

Signed-off-by: JiangJiaWei1103 <[email protected]>

rueian · 2025-12-08T00:13:23Z

The helm lint is failing.

pickymodel · 2025-12-08T00:55:52Z

The helm lint is failing.

Will fix after getting of work, thanks for reviewing!

Future-Outlier · 2025-12-08T10:21:09Z

cc @seanlaii and @win5923 for help. Note that we need to wait until @andrewsykim is back to discuss the API change.

Signed-off-by: JiangJiaWei1103 <[email protected]>

win5923

Hi @JiangJiaWei1103, Can you also update the comment to mention that JobDeploymentStatus is also support?

kuberay/ray-operator/config/samples/ray-job.deletion-rules.yaml

Lines 12 to 22 in e32405e

    
             # DeletionStrategy defines the deletion policies for a RayJob. 
        
             # It allows for fine-grained control over resource cleanup after a job finishes. 
        
             # DeletionRules is a list of deletion rules, processed based on their trigger conditions. 
        
             # While the rules can be used to define a sequence, if multiple rules are overdue (e.g., due to controller downtime), 
        
             # the most impactful rule (e.g., DeleteCluster) will be executed first to prioritize resource cleanup and cost savings. 
        
             deletionStrategy: 
        
               # This sample demonstrates a staged cleanup process for a RayJob. 
        
               # Regardless of whether the job succeeds or fails, the cleanup follows these steps: 
        
               # 1. After 30 seconds, the worker pods are deleted. This allows for quick resource release while keeping the head pod for debugging. 
        
               # 2. After 60 seconds, the entire RayCluster (including the head pod) is deleted. 
        
               # 3. After 90 seconds, the RayJob custom resource itself is deleted, removing it from the Kubernetes API server.

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 · 2025-12-09T11:39:20Z

Hi @JiangJiaWei1103, Can you also update the comment to mention that JobDeploymentStatus is also support?

kuberay/ray-operator/config/samples/ray-job.deletion-rules.yaml

Lines 12 to 22 in e32405e

# DeletionStrategy defines the deletion policies for a RayJob.

# It allows for fine-grained control over resource cleanup after a job finishes.

# DeletionRules is a list of deletion rules, processed based on their trigger conditions.

# While the rules can be used to define a sequence, if multiple rules are overdue (e.g., due to controller downtime),

# the most impactful rule (e.g., DeleteCluster) will be executed first to prioritize resource cleanup and cost savings.

deletionStrategy:

# This sample demonstrates a staged cleanup process for a RayJob.

# Regardless of whether the job succeeds or fails, the cleanup follows these steps:

# 1. After 30 seconds, the worker pods are deleted. This allows for quick resource release while keeping the head pod for debugging.

# 2. After 60 seconds, the entire RayCluster (including the head pod) is deleted.

# 3. After 90 seconds, the RayJob custom resource itself is deleted, removing it from the Kubernetes API server.

Hi @win5923, nice suggestion. I'm considering adding one more sample demonstrating JobDeploymentStatus-based deletion rules, wdyt?

ray-operator/controllers/ray/rayjob_controller.go

ray-operator/controllers/ray/utils/validation.go

ray-operator/apis/ray/v1/rayjob_types.go

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

Signed-off-by: JiangJiaWei1103 <[email protected]>

ray-operator/controllers/ray/utils/validation.go

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: 江家瑋 <[email protected]>

Signed-off-by: JiangJiaWei1103 <[email protected]>

…olicy Signed-off-by: JiangJiaWei1103 <[email protected]>

Signed-off-by: JiangJiaWei1103 <[email protected]>

seanlaii

Thanks! Overall LGTM. Only some reminders:

We might need to either move the e2e tests for DeletionStrategy to a separate action in the CI pipeline or increase the timeout for the e2e tests as it exceeds the current timeout: 40mins. cc @rueian
It might be good to clarify that we evaluate the rules in order, so if a user specifies a different deletionPolicy for a similar status, the first deletionRule will be used, for example:

deletionRules:
    - condition:
        jobStatus: FAILED
        ttlSeconds: 30
      policy: DeleteWorkers
    - condition:
        jobDeploymentStatus: FAILED
        ttlSeconds: 30
      policy: DeleteCluster

Thanks!

ray-operator/apis/ray/v1/rayjob_types.go

JiangJiaWei1103 · 2025-12-14T01:52:21Z

Thanks! Overall LGTM. Only some reminders:

We might need to either move the e2e tests for DeletionStrategy to a separate action in the CI pipeline or increase the timeout for the e2e tests as it exceeds the current timeout: 40mins. cc @rueian

It might be good to clarify that we evaluate the rules in order, so if a user specifies a different deletionPolicy for a similar status, the first deletionRule will be used, for example:
deletionRules:
    - condition:
        jobStatus: FAILED
        ttlSeconds: 30
      policy: DeleteWorkers
    - condition:
        jobDeploymentStatus: FAILED
        ttlSeconds: 30
      policy: DeleteCluster
Thanks!

Thanks for reviewing! For the first one, let's wait for rueian's reply.

As for the second, since both rules match the the corresponding status, they will be added to overdueRules. selectMostImpactfulRule then prioritize the most important rule (DeleteCluster 3 > DeleteWorkers 2 in this case), so I think DeleteCluster will be executed first. Following demonstrates an example:

apiVersion: ray.io/v1
kind: RayJob
metadata:
  namespace: default
  name: demo-del-rules
spec:
  submissionMode: "K8sJobMode"
  entrypoint: "python -c 'import sys, time; time.sleep(45); sys.exit(1)'"

  deletionStrategy:
    deletionRules:
    - condition:
        jobStatus: FAILED
        ttlSeconds: 10
      policy: DeleteWorkers
    - condition:
        jobDeploymentStatus: Failed
        ttlSeconds: 10
      policy: DeleteCluster

    # ...

The following shows the most important policy DeleteCluster is executed:

{"level":"info","ts":"2025-12-14T09:42:29.001+0800","logger":"controllers.RayJob","msg":"Executing the most impactful overdue deletion rule","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"a9595cd2-df62-4291-b8b7-3e47a772ae6e","deletionMechanism":"DeletionRules","rule":{"policy":"DeleteCluster","condition":{"jobDeploymentStatus":"Failed","ttlSeconds":10}},"overdueRulesCount":2}
{"level":"info","ts":"2025-12-14T09:42:29.001+0800","logger":"controllers.RayJob","msg":"Executing deletion policy: DeleteCluster","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"a9595cd2-df62-4291-b8b7-3e47a772ae6e","RayCluster":"del-seq-gcz64"}
{"level":"info","ts":"2025-12-14T09:42:29.013+0800","logger":"controllers.RayJob","msg":"The associated RayCluster for RayJob is deleted","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"a9595cd2-df62-4291-b8b7-3e47a772ae6e","RayCluster":{"name":"del-seq-gcz64","namespace":"default"}}
{"level":"info","ts":"2025-12-14T09:42:29.013+0800","logger":"controllers.RayJob","msg":"deleteClusterResources","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"a9595cd2-df62-4291-b8b7-3e47a772ae6e","isClusterDeleted":false}
{"level":"info","ts":"2025-12-14T09:42:29.013+0800","logger":"controllers.RayJob","msg":"All applicable deletion rules have been processed.","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"a9595cd2-df62-4291-b8b7-3e47a772ae6e","deletionMechanism":"DeletionRules"}
{"level":"info","ts":"2025-12-14T09:42:29.015+0800","logger":"controllers.RayJob","msg":"RayJob","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"7ea2a0cf-1f04-4978-b819-ff7f4784b50e","JobStatus":"FAILED","JobDeploymentStatus":"Failed","SubmissionMode":"K8sJobMode"}
{"level":"info","ts":"2025-12-14T09:42:29.015+0800","logger":"controllers.RayJob","msg":"Skipping completed deletion rule","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"7ea2a0cf-1f04-4978-b819-ff7f4784b50e","deletionMechanism":"DeletionRules","rule":{"policy":"DeleteWorkers","condition":{"jobStatus":"FAILED","ttlSeconds":10}}}
{"level":"info","ts":"2025-12-14T09:42:29.015+0800","logger":"controllers.RayJob","msg":"Skipping completed deletion rule","RayJob":{"name":"del-seq","namespace":"default"},"reconcileID":"7ea2a0cf-1f04-4978-b819-ff7f4784b50e","deletionMechanism":"DeletionRules","rule":{"policy":"DeleteCluster","condition":{"jobDeploymentStatus":"Failed","ttlSeconds":10}}}

If I'm mistaken, please let me know. Thanks a lot!

seanlaii · 2025-12-14T01:54:48Z

As for the second, since both rules match the the corresponding status, they will be added to overdueRules. selectMostImpactfulRule then prioritize the most important rule (DeleteCluster 3 > DeleteWorkers 2 in this case) to execute, so I think DeleteCluster will be executed first.

You are right. My mistake. Forgot that it will first fetch all the rules that are matched. Thanks for the explanation!

seanlaii

Thanks for reviewing! For the first one, let's wait for Rueian's reply.

Actually, the current total time for the RayJob e2e tests is around 38 mins, which doesn't exceed 40 mins. We can increase it if needed in the future. It should be fine for now.

Thanks! LGTM.

Tag @rueian for the second pass. Thanks!

machichima · 2025-12-14T03:03:02Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

+							WithPolicy(rayv1.DeleteNone).
+							WithCondition(rayv1ac.DeletionCondition().
+								WithJobDeploymentStatus(rayv1.JobDeploymentStatusFailed).
+								WithTTLSeconds(10)), // 10 second TTL for testing


Do you think we can set TTL to shorter time here to speed up? As we wait after TTL to do the assertion

This works for me. Using a shorter TTL would allow a faster requeue with a smaller nextRequeueTime. How about setting it to 2?

Hi Nary,

I revert the commit for testing shorter TTL since we need to check resource preservation for a short period of time after the condition is matched. For example:

kuberay/ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

Lines 81 to 94 in 615fa88

g.Consistently(func(gg Gomega) {

cluster, err := GetRayCluster(test, namespace.Name, rayClusterName)

gg.Expect(err).NotTo(HaveOccurred())

gg.Expect(cluster).NotTo(BeNil())

workerPods, err := GetWorkerPods(test, cluster)

gg.Expect(err).NotTo(HaveOccurred())

gg.Expect(workerPods).ToNot(BeEmpty())

headPod, err := GetHeadPod(test, cluster)

gg.Expect(err).NotTo(HaveOccurred())

gg.Expect(headPod).NotTo(BeNil())

jobObj, err := GetRayJob(test, rayJob.Namespace, rayJob.Name)

gg.Expect(err).NotTo(HaveOccurred())

gg.Expect(jobObj).NotTo(BeNil())

}, 8*time.Second, 2*time.Second).Should(Succeed()) // Check every 2s for 8s

machichima · 2025-12-14T03:06:14Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

+			jobObj, err := GetRayJob(test, rayJob.Namespace, rayJob.Name)
+			gg.Expect(err).NotTo(HaveOccurred())
+			gg.Expect(jobObj).NotTo(BeNil())
+			cluster, err := GetRayCluster(test, namespace.Name, rayClusterName)
+			gg.Expect(err).NotTo(HaveOccurred())
+			gg.Expect(cluster).NotTo(BeNil())
+			workerPods, err := GetWorkerPods(test, cluster)
+			gg.Expect(err).NotTo(HaveOccurred())
+			gg.Expect(workerPods).ToNot(BeEmpty())


nit: I saw this check used in multiple places. Maybe we can extract this into a verifyAllResourcesExist function for simplification? WDYT?

Nice suggestion. How about we refactor this part and handle the cleanup logic in a follow-up PR? That way, we can keep this PR more focused and avoid introducing too many changes at once.

machichima · 2025-12-14T03:09:17Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

+		// Cleanup: delete RayJob to release cluster and pods.
+		LogWithTimestamp(test.T(), "Cleaning up RayJob %s/%s after DeleteNone scenario", rayJob.Namespace, rayJob.Name)
+		err = test.Client().Ray().RayV1().RayJobs(rayJob.Namespace).Delete(test.Ctx(), rayJob.Name, metav1.DeleteOptions{})
+		g.Expect(err).NotTo(HaveOccurred())
+		g.Eventually(func() error { _, err := GetRayJob(test, rayJob.Namespace, rayJob.Name); return err }, TestTimeoutMedium).Should(WithTransform(k8serrors.IsNotFound, BeTrue()))
+		g.Eventually(func() error {
+			_, err := GetRayCluster(test, namespace.Name, rayClusterName)
+			return err
+		}, TestTimeoutMedium).Should(WithTransform(k8serrors.IsNotFound, BeTrue()))
+		LogWithTimestamp(test.T(), "Cleanup after DeleteNone scenario complete")
+	})


nit: no need to be in this PR, but I think we can probably extract the cleanup to helper function, then we can put the clean up to defer block after the rayjob creation to make test easier to follow

cc @rueian to confirm if we want to do this

win5923

LGTM! Thank you.

JiangJiaWei1103 · 2025-12-07T23:27:21Z

ray-operator/apis/ray/v1/rayjob_types.go

+	// JobDeploymentStatus is the terminal status of the RayJob deployment that triggers this condition.
+	// For the initial implementation, only "Failed" is supported.
+	// +kubebuilder:validation:Enum=Failed
+	// +optional
+	JobDeploymentStatus *JobDeploymentStatus `json:"jobDeploymentStatus,omitempty"`


For now, we only support Failed option.

ray-operator/controllers/ray/utils/validation.go

ray-operator/controllers/ray/rayjob_controller.go

ray-operator/apis/ray/v1/rayjob_types.go

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

ray-operator/controllers/ray/utils/validation.go

ray-operator/apis/ray/v1/rayjob_types.go

JiangJiaWei1103 · 2025-12-14T23:19:10Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

+							WithPolicy(rayv1.DeleteNone).
+							WithCondition(rayv1ac.DeletionCondition().
+								WithJobDeploymentStatus(rayv1.JobDeploymentStatusFailed).
+								WithTTLSeconds(10)), // 10 second TTL for testing


This works for me. Using a shorter TTL would allow a faster requeue with a smaller nextRequeueTime. How about setting it to 2?

JiangJiaWei1103 · 2025-12-14T23:22:07Z

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go

+			jobObj, err := GetRayJob(test, rayJob.Namespace, rayJob.Name)
+			gg.Expect(err).NotTo(HaveOccurred())
+			gg.Expect(jobObj).NotTo(BeNil())
+			cluster, err := GetRayCluster(test, namespace.Name, rayClusterName)
+			gg.Expect(err).NotTo(HaveOccurred())
+			gg.Expect(cluster).NotTo(BeNil())
+			workerPods, err := GetWorkerPods(test, cluster)
+			gg.Expect(err).NotTo(HaveOccurred())
+			gg.Expect(workerPods).ToNot(BeEmpty())


Nice suggestion. How about we refactor this part and handle the cleanup logic in a follow-up PR? That way, we can keep this PR more focused and avoid introducing too many changes at once.

machichima

Overall LGTM! Thank you

Signed-off-by: JiangJiaWei1103 <[email protected]>

This reverts commit 0588f35. We need to pass consistency checks for resource preservation. Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 added 6 commits December 7, 2025 12:29

feat: Support JobDeploymentStatus as the deletion condition

8c2132a

Signed-off-by: JiangJiaWei1103 <[email protected]>

chore: Regenerate utility codes

596ed83

Signed-off-by: JiangJiaWei1103 <[email protected]>

docs: Update api docs

f0766c4

Signed-off-by: JiangJiaWei1103 <[email protected]>

fix(test): Change JobStatus of the deletion condition from val to ptr

6ced1b6

Signed-off-by: JiangJiaWei1103 <[email protected]>

test: Add JobDeploymentStatus-based e2e tests with four deletion poli…

909fc40

…cies Signed-off-by: JiangJiaWei1103 <[email protected]>

test: Add validation tests for JobDeploymentStatus-based deletion rules

378cc23

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners December 7, 2025 23:26

fix: Sync CRD yaml files into helm chart

1c0cf0b

Signed-off-by: JiangJiaWei1103 <[email protected]>

win5923 reviewed Dec 8, 2025

View reviewed changes

docs: Support JobDeploymentStatus as deletion condition

72ed628

Signed-off-by: JiangJiaWei1103 <[email protected]>

win5923 reviewed Dec 9, 2025

View reviewed changes

ray-operator/controllers/ray/rayjob_controller.go Outdated Show resolved Hide resolved

win5923 reviewed Dec 9, 2025

View reviewed changes

ray-operator/controllers/ray/utils/validation.go Show resolved Hide resolved

win5923 reviewed Dec 9, 2025

View reviewed changes

ray-operator/apis/ray/v1/rayjob_types.go Outdated Show resolved Hide resolved

win5923 reviewed Dec 9, 2025

View reviewed changes

ray-operator/test/e2erayjob/rayjob_deletion_strategy_test.go Show resolved Hide resolved

JiangJiaWei1103 added 2 commits December 10, 2025 07:23

refactor: Add a helper to check rule match

aa8f8c1

Signed-off-by: JiangJiaWei1103 <[email protected]>

docs: Complete TTLSeconds description

1c6c28d

Signed-off-by: JiangJiaWei1103 <[email protected]>

machichima reviewed Dec 10, 2025

View reviewed changes

ray-operator/controllers/ray/utils/validation.go Outdated Show resolved Hide resolved

ray-operator/controllers/ray/utils/validation.go Outdated Show resolved Hide resolved

JiangJiaWei1103 and others added 6 commits December 10, 2025 20:53

refactor: Keep validation logic aligned with kubebuilder

2386ea7

Co-authored-by: Nary Yeh <[email protected]> Signed-off-by: 江家瑋 <[email protected]>

refactor: Write helper for validating deletion condition

1d9d0a4

Signed-off-by: JiangJiaWei1103 <[email protected]>

refactor: Simplify logic for assigning an empty map to track TTL by p…

83eb5eb

…olicy Signed-off-by: JiangJiaWei1103 <[email protected]>

refactor: Simplify deletion condition matching logic

69f28c5

Signed-off-by: JiangJiaWei1103 <[email protected]>

docs: Make deletion rule uniqueness check comment more clear

36c989b

Signed-off-by: JiangJiaWei1103 <[email protected]>

refactor: Use explicit string type to handle both conditions

a3a82ff

Signed-off-by: JiangJiaWei1103 <[email protected]>

JiangJiaWei1103 requested a review from machichima December 12, 2025 13:38

seanlaii reviewed Dec 13, 2025

View reviewed changes

ray-operator/apis/ray/v1/rayjob_types.go Show resolved Hide resolved

seanlaii approved these changes Dec 14, 2025

View reviewed changes

machichima reviewed Dec 14, 2025

View reviewed changes

win5923 approved these changes Dec 14, 2025

View reviewed changes

win5923 approved these changes Dec 14, 2025 •

edited

Loading

View reviewed changes

JiangJiaWei1103 commented Dec 18, 2025

View reviewed changes

machichima approved these changes Dec 18, 2025

View reviewed changes

JiangJiaWei1103 added 2 commits December 18, 2025 21:18

test: Shorten TTL to speed up e2e test

0588f35

Signed-off-by: JiangJiaWei1103 <[email protected]>

Revert "test: Shorten TTL to speed up e2e test"

615fa88

This reverts commit 0588f35. We need to pass consistency checks for resource preservation. Signed-off-by: JiangJiaWei1103 <[email protected]>

rueian approved these changes Dec 19, 2025

View reviewed changes

rueian merged commit 66744cb into ray-project:master Dec 19, 2025
27 checks passed

JiangJiaWei1103 mentioned this pull request Dec 21, 2025

[Refactor] Refactor ray job deletion strategy e2e tests #4294

Open

JiangJiaWei1103 moved this to Done in My Kuberay & Ray Dec 24, 2025

JiangJiaWei1103 added this to My Kuberay & Ray Dec 24, 2025

	# DeletionStrategy defines the deletion policies for a RayJob.
	# It allows for fine-grained control over resource cleanup after a job finishes.
	# DeletionRules is a list of deletion rules, processed based on their trigger conditions.
	# While the rules can be used to define a sequence, if multiple rules are overdue (e.g., due to controller downtime),
	# the most impactful rule (e.g., DeleteCluster) will be executed first to prioritize resource cleanup and cost savings.
	deletionStrategy:
	# This sample demonstrates a staged cleanup process for a RayJob.
	# Regardless of whether the job succeeds or fails, the cleanup follows these steps:
	# 1. After 30 seconds, the worker pods are deleted. This allows for quick resource release while keeping the head pod for debugging.
	# 2. After 60 seconds, the entire RayCluster (including the head pod) is deleted.
	# 3. After 90 seconds, the RayJob custom resource itself is deleted, removing it from the Kubernetes API server.

	g.Consistently(func(gg Gomega) {
	cluster, err := GetRayCluster(test, namespace.Name, rayClusterName)
	gg.Expect(err).NotTo(HaveOccurred())
	gg.Expect(cluster).NotTo(BeNil())
	workerPods, err := GetWorkerPods(test, cluster)
	gg.Expect(err).NotTo(HaveOccurred())
	gg.Expect(workerPods).ToNot(BeEmpty())
	headPod, err := GetHeadPod(test, cluster)
	gg.Expect(err).NotTo(HaveOccurred())
	gg.Expect(headPod).NotTo(BeNil())
	jobObj, err := GetRayJob(test, rayJob.Namespace, rayJob.Name)
	gg.Expect(err).NotTo(HaveOccurred())
	gg.Expect(jobObj).NotTo(BeNil())
	}, 8time.Second, 2time.Second).Should(Succeed()) // Check every 2s for 8s

[Feature] Support JobDeploymentStatus as the deletion condition #4262

[Feature] Support JobDeploymentStatus as the deletion condition #4262

Uh oh!

Conversation

JiangJiaWei1103 commented Dec 7, 2025

Why are these changes needed?

Changes

Implementation Details

Related issue number

Checks

Uh oh!

rueian commented Dec 8, 2025

Uh oh!

pickymodel commented Dec 8, 2025

Uh oh!

Future-Outlier commented Dec 8, 2025

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

JiangJiaWei1103 commented Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

seanlaii left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JiangJiaWei1103 commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seanlaii commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seanlaii left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

seanlaii left a comment •

edited

Loading

JiangJiaWei1103 commented Dec 14, 2025 •

edited

Loading

seanlaii commented Dec 14, 2025 •

edited

Loading

seanlaii left a comment •

edited

Loading