Skip to content

[SPARK-40187][DOCS] Add Apache YuniKorn scheduler docs #37622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yangwwei wants to merge 4 commits into from
Closed

[SPARK-40187][DOCS] Add Apache YuniKorn scheduler docs #37622

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a7a1ad1
[SPARK-40187] Add doc for using Apache YuniKorn as a customized sched…
yangwwei Aug 23, 2022
fb50eab
minor updates
yangwwei Aug 23, 2022
9dd97fb
address review comments
yangwwei Aug 23, 2022
4cf1a1d
address review comments
yangwwei Aug 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1811,6 +1811,50 @@ spec:
queue: default
```

#### Using Apache YuniKorn as Customized Scheduler for Spark on Kubernetes

[Apache YuniKorn](https://yunikorn.apache.org/) is a resource scheduler for Kubernetes that provides advanced batch scheduling
capabilities, such as job queuing, resource fairness, min/max queue capacity and flexible job ordering policies.
For available Apache YuniKorn features, please refer to [this doc](https://yunikorn.apache.org/docs/next/get_started/core_features).

@dongjoon-hyun dongjoon-hyun Aug 23, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We cannot use the version number alias next here because it will be fragile in the future.

We need Apache YuniKorn community's help here, @yangwwei .

@yangwwei yangwwei Aug 23, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @dongjoon-hyun This is how the doc site works, next -> is the current under-development version, we shouldn't use this, that's a good point; but I think we can use the latest stable version: this points to https://yunikorn.apache.org/docs/. Only the past versions are accessible via https://yunikorn.apache.org/docs/{VERSION_NUM}, that's why you did not see 1.0.0 there, 1.0.0 is the current stable version.

If we use a hard-coded version, e.g 1.0.0 here, we will need to come back to update the doc quite often, I don't feel that is good. So my question is: is it better to use the latest stable version here or a hard-coded version that will need updates over time? Please let me know, thanks!

@dongjoon-hyun dongjoon-hyun Aug 23, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I asked here. Apache YuniKorn community should provide 1.0.0 like Apache Spark did.

That is mandatory in order to guarantee when we support something. Please see Volcano example.

we will need to come back to update the doc quite often,


##### Prerequisites
Comment thread
yangwwei marked this conversation as resolved.

Install Apache YuniKorn:

```bash
helm repo add yunikorn https://apache.github.io/yunikorn-release
helm repo update
kubectl create namespace yunikorn
helm install yunikorn yunikorn/yunikorn --namespace yunikorn
Comment thread
yangwwei marked this conversation as resolved.
Outdated
```

the above steps will install the latest version of YuniKorn on an existing Kubernetes cluster.

@dongjoon-hyun dongjoon-hyun Aug 23, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use specific version instead recommending the latest version in the doc.

@dongjoon-hyun dongjoon-hyun Aug 23, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the above -> The above


##### Get started

Submit Spark jobs with the following extra options:

```bash
--conf spark.kubernetes.scheduler.name=yunikorn
--conf spark.kubernetes.driver.annotation.yunikorn.apache.org/app-id={{APP_ID}}
--conf spark.kubernetes.executor.annotation.yunikorn.apache.org/app-id={{APP_ID}}
```

Note, `{{APP_ID}}` is the builtin variable that will be substituted with Spark job ID automatically.
Comment thread
yangwwei marked this conversation as resolved.
Outdated
With the above configuration, the job will be scheduled by YuniKorn scheduler instead of the default Kubernetes scheduler.

##### Work with YuniKorn queues
Comment thread
yangwwei marked this conversation as resolved.
Outdated

Apache YuniKorn supports 2 types of resource queues:

- Static
- Dynamic

The static queues are predefined in YuniKorn configmap, and the dynamic queues are automatically created by the scheduler
based on [placement rules](https://yunikorn.apache.org/docs/next/user_guide/placement_rules). Spark supports to run with
Comment thread
yangwwei marked this conversation as resolved.
Outdated
both queue setup. Refer to this [doc](https://yunikorn.apache.org/docs/next/user_guide/resource_quota_management) for more
information about how to run Spark with different queue setup.

Comment thread
yangwwei marked this conversation as resolved.
### Stage Level Scheduling Overview

Stage level scheduling is supported on Kubernetes when dynamic allocation is enabled. This also requires <code>spark.dynamicAllocation.shuffleTracking.enabled</code> to be enabled since Kubernetes doesn't support an external shuffle service at this time. The order in which containers for different profiles is requested from Kubernetes is not guaranteed. Note that since dynamic allocation on Kubernetes requires the shuffle tracking feature, this means that executors from previous stages that used a different ResourceProfile may not idle timeout due to having shuffle data on them. This could result in using more cluster resources and in the worst case if there are no remaining resources on the Kubernetes cluster then Spark could potentially hang. You may consider looking at config <code>spark.dynamicAllocation.shuffleTracking.timeout</code> to set a timeout, but that could result in data having to be recomputed if the shuffle data is really needed.
Expand Down