[SPARK-40187][DOCS] Add Apache YuniKorn scheduler docs

[yangwwei]

What changes were proposed in this pull request?

Add a section under customized-kubernetes-schedulers-for-spark-on-kubernetes to explain how to run Spark with Apache YuniKorn. This is based on the review comments from #35663.

Why are the changes needed?

Explain how to run Spark with Apache YuniKorn

Does this PR introduce any user-facing change?

No

[dongjoon-hyun]

We cannot use the version number alias next here because it will be fragile in the future.

• We need to use a concrete version v1.0.0 link instead.
• However, Apache YuniKorn doesn't provide 1.0.0 yet.
  • https://yunikorn.apache.org/docs/1.0.0/ is broken.
  • The only latest version seems to be https://yunikorn.apache.org/docs/0.12.2/ .

We need Apache YuniKorn community's help here, @yangwwei .

[yangwwei]

hi @dongjoon-hyun This is how the doc site works, next -> is the current under-development version, we shouldn't use this, that's a good point; but I think we can use the latest stable version: this points to https://yunikorn.apache.org/docs/. Only the past versions are accessible via https://yunikorn.apache.org/docs/{VERSION_NUM}, that's why you did not see 1.0.0 there, 1.0.0 is the current stable version.

If we use a hard-coded version, e.g 1.0.0 here, we will need to come back to update the doc quite often, I don't feel that is good. So my question is: is it better to use the latest stable version here or a hard-coded version that will need updates over time? Please let me know, thanks!

[dongjoon-hyun]

That's what I asked here. Apache YuniKorn community should provide 1.0.0 like Apache Spark did.

• https://spark.apache.org/docs/3.3.0/

That is mandatory in order to guarantee when we support something. Please see Volcano example.

we will need to come back to update the doc quite often,

[dongjoon-hyun]

Please use specific version instead recommending the latest version in the doc.

[dongjoon-hyun]

the above -> The above

[dongjoon-hyun]

Thank you for making a PR, @yangwwei . I reviewed and will hold on this PR to align with other validation.

[AmplabJenkins]

Can one of the admins verify this patch?

[dongjoon-hyun]

It seems that I wasn't clear enough to you. We need a specific version number, @yangwwei . -1 for adding a doc without version version.

https://yunikorn.apache.org/docs/get_started/core_features.

[yangwwei]

It seems that I wasn't clear enough to you. We need a specific version number, @yangwwei . -1 for adding a doc without version version.

https://yunikorn.apache.org/docs/get_started/core_features.

You are absolutely clear. Sure, let me find out to get this supported with our documentation framework. Thanks

[yangwwei]

@dongjoon-hyun could you please take a look at the updated version? Here are the changes:

1. Used 1.0.0 in the installation example to comply with Spark community requirements
2. Removed the "Work with YuniKorn queues" as that adds complexity to the doc, and for Spark users, that's not something that has to be done
3. Addressed other review comments

there is one more thing about our version doc. I have implemented some temp workaround in https://issues.apache.org/jira/browse/YUNIKORN-1293, but YuniKorn community folks made a very good point that the workaround wasn't sustainable. Since now there is only the following links in the doc:

• https://yunikorn.apache.org/
• https://yunikorn.apache.org/docs/get_started/core_features

I think it should be fine that not bind to a specific version, what do you think? YuniKorn folks also pointed out that there are similar docs like the following already:

Volcano feature steps help users to create a Volcano PodGroup and set driver/executor pod annotation to link with this [PodGroup](https://volcano.sh/en/docs/podgroup/).

and

Volcano defines PodGroup spec using [CRD yaml](https://volcano.sh/en/docs/podgroup/#example). 

please let me know your thought for this, thanks!

[dongjoon-hyun]

In that case, why don't you add Apache Spark Support Matrix in Apache YuniKorn page?

I think it should be fine that not bind to a specific version, what do you think?

Specifically, I'm suggesting the following.

• Apache Spark simply adds a forward link to Apache YuniKorn's Support Matrix page.
• Apache YuniKorn page (v1.0.0 or the future release) will test and maintain it from YuniKorn community side.

Apache Spark does that for Arrow project's PyArrow support.

• https://spark.apache.org/docs/3.3.0/

[dongjoon-hyun]

BTW, let me test YuniKorn according to this doc tomorrow further, @yangwwei .

[dongjoon-hyun]

According to apache/yunikorn-site#180 (comment) , I understand the context (including rollback) and let's hold on this until next week (YuniKorn v1.1).

[yangwwei]

Hi @dongjoon-hyun

Please see my thoughts w.r.t this comment

In that case, why don't you add Apache Spark Support Matrix to Apache YuniKorn page?

I don't think that will be necessary. Apache YuniKorn is a scheduler, a replacement to the default scheduler, it isn't so sensitive about Spark versions. Any Spark can run on YuniKorn with some necessary configs. In the recent version with the support of https://issues.apache.org/jira/browse/SPARK-38383, submitting jobs to YuniKorn is even easier (what was introduced in this doc). This is like we do not need a Spark support matrix in YARN, or Kubernetes.

Adding such a matrix on the YuniKorn side is cumbersome, we probably will need to list all Spark versions, which isn't useful for the end users.

[yangwwei]

According to apache/yunikorn-site#180 (comment) , I understand the context (including rollback) and let's hold on this until next week (YuniKorn v1.1).

Please take a look at the updated version. I do not think we depend on the 1.0.0 doc link anymore. We just make sure we explicitly set the version in the installation example. For general docs, we can point to https://yunikorn.apache.org.

[dongjoon-hyun]

Got it. Let me think about that again from Apache Spark user perspective. I believe we can find some sweet spots where both communities satisfy, @yangwwei . Thank you again for all your contribution and collaboration.

[yangwwei]

@dongjoon-hyun circle back on this. does the latest version look good to you?
Anything else you want me to address in this doc? Please let me know, thanks!

[dongjoon-hyun]

+1, LGTM. Thank you, @yangwwei . I also tested all existing tests with YuniKorn scheduler.
Merged to master/3.3.

[dongjoon-hyun]

Sorry for being late, @yangwwei . We can move forward more based on this.

[dongjoon-hyun]

I created a test suite PR, @yangwwei .

• [SPARK-40302][K8S][TESTS] Add YuniKornSuite #37753

[yangwwei]

Hi, @dongjoon-hyun thanks a lot for helping on this.
This is a great community collaboration between YuniKorn and Spark, thank you so much!

[dongjoon-hyun]

Thank YOU, @yangwwei .