[KYUUBI #6686] Ignore Spark pod container state if pod is terminated #6690

turboFei · 2024-09-12T04:43:18Z

🔍 Description

Issue References 🔗

To close #6686

The pod already in failed state, and the driver container is in waiting state.

We shall mark the application terminated and ignore the container state.

Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklist 📝

This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala

codecov-commenter · 2024-09-12T05:55:25Z

Codecov Report

Attention: Patch coverage is 0% with 16 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (db5ce0c) to head (0d4c8a2).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...kyuubi/engine/KubernetesApplicationOperation.scala	0.00%	15 Missing ⚠️
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	0.00%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6690   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         684     684           
  Lines       42233   42244   +11     
  Branches     5755    5756    +1     
======================================
- Misses      42233   42244   +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

turboFei · 2024-09-14T19:29:10Z

thanks, merged to 1.0.0.

…pp state than terminated pod state ### Why are the changes needed? I found that, for a kyuubi batch on kubernetes. 1. It has been `FINISHED`. 2. then I delete the pod manually, then I check the k8s-audit.log, then the appState became `FAILED`. ``` 2025-04-15 11:16:30.453 INFO [-675216314-pool-44-thread-839] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: label=61e7d8c1-e5a9-46cd-83e7-c611003f0224 context=97 namespace=dls-prod pod=kyuubi-spark-61e7d8c1-e5a9-46cd-83e7-c611003f0224-driver podState=Running containers=[microvault->ContainerState(running=ContainerStateRunning(startedAt=2025-04-15T18:13:48Z, additionalProperties={}), terminated=null, waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://72704f8e7ccb5e877c8f6b10bf6ad810d0c019e07e0cb5975be733e79762c1ec, exitCode=0, finishedAt=2025-04-15T18:14:22Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T18:13:49Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-228c62e0dc37402bacac189d01b871e4 appState=FINISHED appError='' :2025-04-15 11:16:30.854 INFO [-675216314-pool-44-thread-840] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: label=61e7d8c1-e5a9-46cd-83e7-c611003f0224 context=97 namespace=dls-prod pod=kyuubi-spark-61e7d8c1-e5a9-46cd-83e7-c611003f0224-driver podState=Failed containers=[microvault->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://91654e3ee74e2c31218e14be201b50a4a604c2ad15d3afd84dc6f620e59894b7, exitCode=2, finishedAt=2025-04-15T18:16:30Z, message=null, reason=Error, signal=null, startedAt=2025-04-15T18:13:48Z, additionalProperties={}), waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://72704f8e7ccb5e877c8f6b10bf6ad810d0c019e07e0cb5975be733e79762c1ec, exitCode=0, finishedAt=2025-04-15T18:14:22Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T18:13:49Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-228c62e0dc37402bacac189d01b871e4 appState=FAILED appError='{ ``` This PR is a followup for #6690 , which ignore the container state if POD is terminated. It is more reasonable to respect the terminated container state than terminated pod state. ### How was this patch tested? Integration testing. ``` :2025-04-15 13:53:24.551 INFO [-1077768163-pool-36-thread-3] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: eventType=DELETE label=e0eb4580-3cfa-43bf-bdcc-efeabcabc93c context=97 namespace=dls-prod pod=kyuubi-spark-e0eb4580-3cfa-43bf-bdcc-efeabcabc93c-driver podState=Failed containers=[microvault->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://66c42206730950bd422774e3c1b0f426d7879731788cea609bbfe0daab24a763, exitCode=2, finishedAt=2025-04-15T20:53:22Z, message=null, reason=Error, signal=null, startedAt=2025-04-15T20:52:00Z, additionalProperties={}), waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://9179a73d9d9e148dcd9c13ee6cc29dc3e257f95a33609065e061866bb611cb3b, exitCode=0, finishedAt=2025-04-15T20:52:28Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T20:52:01Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-578df0facbfd4958a07f8d1ae79107dc appState=FINISHED appError='' ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7025 from turboFei/container_terminated. Closes #7025 Closes #6686 a3b2a5a [Wang, Fei] comments 4356d1b [Wang, Fei] fix the app state logical Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]>

…pp state than terminated pod state ### Why are the changes needed? I found that, for a kyuubi batch on kubernetes. 1. It has been `FINISHED`. 2. then I delete the pod manually, then I check the k8s-audit.log, then the appState became `FAILED`. ``` 2025-04-15 11:16:30.453 INFO [-675216314-pool-44-thread-839] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: label=61e7d8c1-e5a9-46cd-83e7-c611003f0224 context=97 namespace=dls-prod pod=kyuubi-spark-61e7d8c1-e5a9-46cd-83e7-c611003f0224-driver podState=Running containers=[microvault->ContainerState(running=ContainerStateRunning(startedAt=2025-04-15T18:13:48Z, additionalProperties={}), terminated=null, waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://72704f8e7ccb5e877c8f6b10bf6ad810d0c019e07e0cb5975be733e79762c1ec, exitCode=0, finishedAt=2025-04-15T18:14:22Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T18:13:49Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-228c62e0dc37402bacac189d01b871e4 appState=FINISHED appError='' :2025-04-15 11:16:30.854 INFO [-675216314-pool-44-thread-840] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: label=61e7d8c1-e5a9-46cd-83e7-c611003f0224 context=97 namespace=dls-prod pod=kyuubi-spark-61e7d8c1-e5a9-46cd-83e7-c611003f0224-driver podState=Failed containers=[microvault->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://91654e3ee74e2c31218e14be201b50a4a604c2ad15d3afd84dc6f620e59894b7, exitCode=2, finishedAt=2025-04-15T18:16:30Z, message=null, reason=Error, signal=null, startedAt=2025-04-15T18:13:48Z, additionalProperties={}), waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://72704f8e7ccb5e877c8f6b10bf6ad810d0c019e07e0cb5975be733e79762c1ec, exitCode=0, finishedAt=2025-04-15T18:14:22Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T18:13:49Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-228c62e0dc37402bacac189d01b871e4 appState=FAILED appError='{ ``` This PR is a followup for #6690 , which ignore the container state if POD is terminated. It is more reasonable to respect the terminated container state than terminated pod state. ### How was this patch tested? Integration testing. ``` :2025-04-15 13:53:24.551 INFO [-1077768163-pool-36-thread-3] org.apache.kyuubi.engine.KubernetesApplicationAuditLogger: eventType=DELETE label=e0eb4580-3cfa-43bf-bdcc-efeabcabc93c context=97 namespace=dls-prod pod=kyuubi-spark-e0eb4580-3cfa-43bf-bdcc-efeabcabc93c-driver podState=Failed containers=[microvault->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://66c42206730950bd422774e3c1b0f426d7879731788cea609bbfe0daab24a763, exitCode=2, finishedAt=2025-04-15T20:53:22Z, message=null, reason=Error, signal=null, startedAt=2025-04-15T20:52:00Z, additionalProperties={}), waiting=null, additionalProperties={}),spark-kubernetes-driver->ContainerState(running=null, terminated=ContainerStateTerminated(containerID=containerd://9179a73d9d9e148dcd9c13ee6cc29dc3e257f95a33609065e061866bb611cb3b, exitCode=0, finishedAt=2025-04-15T20:52:28Z, message=null, reason=Completed, signal=null, startedAt=2025-04-15T20:52:01Z, additionalProperties={}), waiting=null, additionalProperties={})] appId=spark-578df0facbfd4958a07f8d1ae79107dc appState=FINISHED appError='' ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7025 from turboFei/container_terminated. Closes #7025 Closes #6686 a3b2a5a [Wang, Fei] comments 4356d1b [Wang, Fei] fix the app state logical Authored-by: Wang, Fei <[email protected]> Signed-off-by: Wang, Fei <[email protected]> (cherry picked from commit 7e199d6) Signed-off-by: Wang, Fei <[email protected]>

check pod terminated

d60b901

github-actions bot added the module:server label Sep 12, 2024

turboFei changed the title ~~[KYUUBI #6686] Check whether pod based application state terminated~~ [KYUUBI #6686] Mark application terminated if pod is terminated Sep 12, 2024

turboFei self-assigned this Sep 12, 2024

turboFei changed the title ~~[KYUUBI #6686] Mark application terminated if pod is terminated~~ [KYUUBI #6686] Ignore container state if pod is terminated Sep 12, 2024

turboFei changed the title ~~[KYUUBI #6686] Ignore container state if pod is terminated~~ [KYUUBI #6686] Ignore Spark pod container state if pod is terminated Sep 12, 2024

pan3793 reviewed Sep 12, 2024

View reviewed changes

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala Show resolved Hide resolved

github-actions bot added kind:documentation Documentation is a feature! module:common labels Sep 12, 2024

turboFei force-pushed the pod_state branch from ff4255d to 2f57f48 Compare September 12, 2024 06:10

comments

0d4c8a2

turboFei force-pushed the pod_state branch from 2f57f48 to 0d4c8a2 Compare September 12, 2024 06:10

pan3793 approved these changes Sep 12, 2024

View reviewed changes

cxzl25 approved these changes Sep 12, 2024

View reviewed changes

turboFei closed this in f8431da Sep 14, 2024

turboFei added this to the v1.10.0 milestone Sep 14, 2024

turboFei deleted the pod_state branch September 14, 2024 19:28

turboFei mentioned this pull request Apr 15, 2025

[KYUUBI #6686][FOLLOWUP] Prefer terminated container app state than terminated pod state #7025

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[KYUUBI #6686] Ignore Spark pod container state if pod is terminated #6690

[KYUUBI #6686] Ignore Spark pod container state if pod is terminated #6690

Uh oh!

turboFei commented Sep 12, 2024 •

edited

Loading

Uh oh!

Uh oh!

codecov-commenter commented Sep 12, 2024 •

edited

Loading

Uh oh!

turboFei commented Sep 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[KYUUBI #6686] Ignore Spark pod container state if pod is terminated #6690

[KYUUBI #6686] Ignore Spark pod container state if pod is terminated #6690

Uh oh!

Conversation

turboFei commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklist 📝

Uh oh!

Uh oh!

codecov-commenter commented Sep 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

turboFei commented Sep 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

turboFei commented Sep 12, 2024 •

edited

Loading

codecov-commenter commented Sep 12, 2024 •

edited

Loading