[SPARK-36058][K8S] Add support for statefulset APIs in K8s #33508

holdenk · 2021-07-23T23:56:57Z

What changes were proposed in this pull request?

Generalize the pod allocator and add support for statefulsets.

Why are the changes needed?

Allocating individual pods in Spark can be not ideal for some clusters and using higher level operators like statefulsets and replicasets can be useful.

Does this PR introduce any user-facing change?

Yes new config options.

How was this patch tested?

Completed: New unit & basic integration test
PV integration tests

SparkQA · 2021-07-24T00:11:29Z

Test build #141589 has finished for PR 33508 at commit 4ec47f1.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala

examples/src/main/scala/org/apache/spark/examples/MiniReadWriteTest.scala

SparkQA · 2021-07-24T00:50:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46106/

SparkQA · 2021-07-24T01:26:17Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46106/

SparkQA · 2021-07-28T19:46:21Z

Test build #141784 has finished for PR 33508 at commit 17b3c1b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-28T20:22:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46298/

SparkQA · 2021-07-28T20:25:09Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46297/

SparkQA · 2021-07-28T20:58:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46298/

SparkQA · 2021-07-28T21:48:56Z

Test build #141785 has finished for PR 33508 at commit 168082e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala

dongjoon-hyun · 2021-07-29T18:25:19Z

...netes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/AbstractPodsAllocator.scala

cc @shrutig since this is another private[spark] and will change ExecutorPodsAllocator here.

...rnetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

SparkQA · 2021-07-29T19:13:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46365/

SparkQA · 2021-07-29T19:46:52Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46365/

SparkQA · 2021-07-29T21:01:57Z

Test build #141854 has finished for PR 33508 at commit fb0c010.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-30T00:45:24Z

Test build #141861 has finished for PR 33508 at commit 8ceeef6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-30T00:54:24Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46372/

SparkQA · 2021-07-30T02:25:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46375/

SparkQA · 2021-07-30T03:02:50Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46375/

SparkQA · 2021-07-30T04:03:25Z

Test build #141864 has finished for PR 33508 at commit db8f593.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh

SparkQA · 2021-08-23T20:15:05Z

Test build #142705 has finished for PR 33508 at commit 0502e19.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kbendick

A few nits I'll leave at your discretion, but overall this looks good to me for supporting statefulsets and other executor allocation strategies. Would love to get this in to make testing the use of this API easier. 🙂

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

...es/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala

.../core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocatorSuite.scala

...ation-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/BasicTestsSuite.scala

…nd fix test name

SparkQA · 2021-08-24T00:30:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47211/

…-like-things

SparkQA · 2021-08-24T01:25:54Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47211/

SparkQA · 2021-08-24T02:13:43Z

Test build #142710 has finished for PR 33508 at commit a638719.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-24T02:31:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47212/

SparkQA · 2021-08-24T03:27:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47212/

holdenk · 2021-08-24T05:01:26Z

jenkins retest this please.

SparkQA · 2021-08-24T05:51:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47215/

SparkQA · 2021-08-24T06:47:37Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47215/

SparkQA · 2021-08-24T07:36:53Z

Test build #142715 has finished for PR 33508 at commit 4f3c0cc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class Aggregation implements Serializable
public final class Count implements AggregateFunc
public final class CountStar implements AggregateFunc
public final class Max implements AggregateFunc
public final class Min implements AggregateFunc
public final class Sum implements AggregateFunc

SparkQA · 2021-08-24T10:09:43Z

Test build #142711 has finished for PR 33508 at commit 4f3c0cc.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class Aggregation implements Serializable
public final class Count implements AggregateFunc
public final class CountStar implements AggregateFunc
public final class Max implements AggregateFunc
public final class Min implements AggregateFunc
public final class Sum implements AggregateFunc

…-like-things

SparkQA · 2021-08-25T21:37:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47273/

SparkQA · 2021-08-25T22:18:19Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47273/

holdenk · 2021-08-26T00:39:58Z

Merged to the current dev branch (targetting 3.3)

SparkQA · 2021-08-26T05:17:25Z

Test build #142773 has finished for PR 33508 at commit 5be1942.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

yipen · 2022-02-28T03:04:37Z

@holdenk I noticed that there already set ownerreference between the executor pods and driver pod. And there also set ownerreference between the statefulset and dirver. Not sure if these are duplicated?

…ption correctly ### What changes were proposed in this pull request? This PR aims to fix error message to include the exception because #33508 missed the string interpolation prefix, `s"`. https://github.com/apache/spark/blob/c032928515e74367137c668ce692d8fd53696485/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala#L110 ### Why are the changes needed? To show the intended message. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. Closes #35829 from dongjoon-hyun/SPARK-36058. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Yuming Wang <[email protected]>

### What changes were proposed in this pull request? This PR aims to fix RBAC to allow `Spark` driver to create `StatefulSet`. ### Why are the changes needed? We need to fix this to allow Apache Spark's `StatefulSetPodsAllocator` which was introduced at Apache Spark 3.3.0. - apache/spark#33508 ### Does this PR introduce _any_ user-facing change? No, this is an additional permission. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #389 from dongjoon-hyun/SPARK-53909. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? Adds support for K8s `Deployment` API to allocate pods. ### Why are the changes needed? Allocating individual pods is not ideal, and we can allocate with higher level APIs. #33508 helps this by adding an interface for arbitrary allocators and adds a statefulset allocator. However, dynamic allocation only works if you have implemented a PodDisruptionBudget associated with the decommission label. Since Deployment uses ReplicaSet, which supports `pod-deletion-cost` annotation, we can avoid needing to create a separate PDB resource, and allow dynamic allocation (w/ shuffle tracking) by adding a low deletion cost to executors we are scaling down. When we scale the Deployment, it will choose to scale down the pods with the low deletion cost. ### Does this PR introduce _any_ user-facing change? Yes, adds user-facing configs ``` spark.kubernetes.executor.podDeletionCost ``` ### How was this patch tested? New unit tests + passing existing unit tests + tested in a cluster with shuffle tracking and dynamic allocation enabled ### Was this patch authored or co-authored using generative AI tooling? No Closes #52867 from ForVic/dev/victors/deployment_allocator. Lead-authored-by: Victor Sunderland <[email protected]> Co-authored-by: victors-oai <[email protected]> Co-authored-by: Victor Sunderland <[email protected]> Signed-off-by: Chao Sun <[email protected]>

### What changes were proposed in this pull request? Adds support for K8s `Deployment` API to allocate pods. ### Why are the changes needed? Allocating individual pods is not ideal, and we can allocate with higher level APIs. apache#33508 helps this by adding an interface for arbitrary allocators and adds a statefulset allocator. However, dynamic allocation only works if you have implemented a PodDisruptionBudget associated with the decommission label. Since Deployment uses ReplicaSet, which supports `pod-deletion-cost` annotation, we can avoid needing to create a separate PDB resource, and allow dynamic allocation (w/ shuffle tracking) by adding a low deletion cost to executors we are scaling down. When we scale the Deployment, it will choose to scale down the pods with the low deletion cost. ### Does this PR introduce _any_ user-facing change? Yes, adds user-facing configs ``` spark.kubernetes.executor.podDeletionCost ``` ### How was this patch tested? New unit tests + passing existing unit tests + tested in a cluster with shuffle tracking and dynamic allocation enabled ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#52867 from ForVic/dev/victors/deployment_allocator. Lead-authored-by: Victor Sunderland <[email protected]> Co-authored-by: victors-oai <[email protected]> Co-authored-by: Victor Sunderland <[email protected]> Signed-off-by: Chao Sun <[email protected]>

github-actions bot added EXAMPLES KUBERNETES labels Jul 23, 2021

dongjoon-hyun reviewed Jul 24, 2021

View reviewed changes

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Jul 24, 2021

View reviewed changes

examples/src/main/scala/org/apache/spark/examples/MiniReadWriteTest.scala Outdated Show resolved Hide resolved

holdenk force-pushed the SPARK-36058-support-replicasets-or-job-api-like-things branch from 67c5e48 to 17b3c1b Compare July 28, 2021 19:06

holdenk changed the title ~~[WIP][SPARK-36058][K8S] Add support for statefulset APIs in K8s~~ [SPARK-36058][K8S] Add support for statefulset APIs in K8s Jul 28, 2021

github-actions bot added the CORE label Jul 28, 2021