Skip to content

[SPARK-38561][K8S][DOCS] Add doc for Customized Kubernetes Schedulers #35869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Yikun wants to merge 2 commits into from
Closed

[SPARK-38561][K8S][DOCS] Add doc for Customized Kubernetes Schedulers #35869

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
eb59337
Add doc for custom sche
Yikun Mar 16, 2022
6aadfea
Apply suggestions from martin
Yikun Mar 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1722,6 +1722,25 @@ spec:
image: will-be-overwritten
```

#### Customized Kubernetes Schedulers for Spark on Kubernetes

Spark allows users to specify a customized scheduler as Spark on Kubernetes scheduler.
Comment thread
Yikun marked this conversation as resolved.
Outdated

1. Specify scheduler name.

Users can specify customized scheduler using <code>spark.kubernetes.scheduler.name</code> or
Comment thread
Yikun marked this conversation as resolved.
Outdated
<code>spark.kubernetes.{driver/executor}.scheduler.name</code> configuration.

2. Specify scheduler related configurations.

Users can use [Pod template](#pod-template), existing configurations to specify label (<code>spark.kubernetes.{driver,executor}.label.*</code>), annotations style (<code>spark.kubernetes.{driver/executor}.annotation.*</code>) scheduler hints.
Comment thread
Yikun marked this conversation as resolved.
Outdated

3. Specify scheduler feature step.

Users may also consider to use <code>spark.kubernetes.{driver/executor}.pod.featureSteps</code> to support more complex requirements and more centralized scheduler hints configure, included but not limited to:
Comment thread
Yikun marked this conversation as resolved.
Outdated
- Creating a scheduler needed additional Kubernetes custom resource for driver/executor scheduling.
Comment thread
Yikun marked this conversation as resolved.
Outdated
- Setting scheduler hints according to configuration or existing Pod info dynamically.
Comment thread
Yikun marked this conversation as resolved.
Outdated

### Stage Level Scheduling Overview

Stage level scheduling is supported on Kubernetes when dynamic allocation is enabled. This also requires <code>spark.dynamicAllocation.shuffleTracking.enabled</code> to be enabled since Kubernetes doesn't support an external shuffle service at this time. The order in which containers for different profiles is requested from Kubernetes is not guaranteed. Note that since dynamic allocation on Kubernetes requires the shuffle tracking feature, this means that executors from previous stages that used a different ResourceProfile may not idle timeout due to having shuffle data on them. This could result in using more cluster resources and in the worst case if there are no remaining resources on the Kubernetes cluster then Spark could potentially hang. You may consider looking at config <code>spark.dynamicAllocation.shuffleTracking.timeout</code> to set a timeout, but that could result in data having to be recomputed if the shuffle data is really needed.
Expand Down