[SPARK-38621][K8S][BUILD][INFRA] Add volcano module to release-build.sh

[Yikun]

What changes were proposed in this pull request?

Add volcano module into release-build.sh

Why are the changes needed?

Add volcano module into release-build.sh

Does this PR introduce any user-facing change?

No

How was this patch tested?

ci passed

[Yikun]

We added volcano module before, and will introduced spark k8s with volcano support in 3.3.

@dongjoon-hyun @HyukjinKwon @holdenk I'm not very familar with release process, but I think we need this PR to make sure volcano module can be published in final release bin/tar?

[Yikun]

Any comments?

[dongjoon-hyun]

For Apache Spark 3.3, -1 from my side. And, even for master branch, since some release managers still uses master branch release script, I believe we need to hold on this until Apache Spark 3.3.0 is released.

If you don't mind, let's talk later for Apache Spark 3.4 after 3.3.0 release.

[dongjoon-hyun]

To reviewers. It was intentional not to expose this module into the Apache Spark release artifacts.

While I've been helping this SPIP, there were many issues in Volcano library and volcano module until the last day of branch cut. I'm also not sure about the stability yet. This module and configurations are re-designed to isolate each custom scheduler from the others. This is still in an experiment stage and I believe we need more time in this area. AFAIK, we couldn't add Yunikorn due to the lack of community bandwidth. Hence, we should not include this into binary artifacts in Apache Spark 3.3. After stabilizing more, we can add it at Apache Spark 3.4 officially.

[Yikun]

@dongjoon-hyun

there were many issues in Volcano library and volcano module until the last day of branch cut.

To be clear, most of these problems because volcano branch selection, recently, we are also continuously verifying the work of volcano and spark, it's stable now.

This is still in an experiment stage and I believe we need more time in this area.

Yes, for sure, we specifically say this in the documentation. For users who want use this experimental feature, what's the best practice? Sounds like we need to mention build release with -Pvolcano in #35870 ?

AFAIK, we couldn't add Yunikorn due to the lack of community bandwidth.

In fact, I have been getting in touch with @yangwwei and working together to make yunikorn integrated better with spark. I will also keeping help yunikorn and spark for further integration in future to make community bandwidth expand a little bit.

[dongjoon-hyun]

No, there was no available release version, wasn't it. Apache Spark 3.3.0 consumes a set of specific released versions, not branch, @Yikun

To be clear, these problems were most issues because volcano branch selection.

For Yunikorn, thanks. I believe they can follow our latest architecture easily if they start.

[dongjoon-hyun]

I know that Volcano also has a long dev history. That's the reason why I was surprised during supporting you. We don't know what we are going to hit more in the future until we try. I'm expecting that Apache Spark 3.3.0 will give us more broader and world-wide feedback and help us. What we need is a safe step to introduce a new feature.

we are also continuously verifying the work of volcano and spark, it's stable now.

[Yikun]

I'm expecting that Apache Spark 3.3.0 will give us more broader and world-wide feedback and help us.

@dongjoon-hyun Totally, agree. then I need to added the some doc like build with -Pvolcano in #35870 .

To be a double confirmed, this doc PR can be backport to 3.3 after getting merged, right? To make sure users can know how to use this experimental features in 3.3.

[dongjoon-hyun]

Yep, that's right. And, it's up to the author and the committer of that PR. You can ping her.

[dongjoon-hyun]

BTW, your previous doc PR is also not in branch-3.3, isn't it?

[Yikun]

BTW, your previous doc PR is also not in branch-3.3, isn't it?

Yep, looks like it is: the #35869 is only landing on master, could you help it? Or I can submit a PR to branch-3.3 (backport to 3.3: #35955).

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!