[SPARK-30371][K8S] Add spark.kubernetes.driver.master conf by wackxu · Pull Request #27029 · apache/spark

wackxu · 2019-12-28T09:34:58Z

What changes were proposed in this pull request?

make KUBERNETES_MASTER_INTERNAL_URL configurable

Why are the changes needed?

we do not always use the default port number 443 to access our kube-apiserver, and even in some mulit-tenant cluster, people do not use the service kubernetes.default.svc to access the kube-apiserver, so make the internal master configurable is necessary。

Does this PR introduce any user-facing change?

user can configure the internal master url by

--conf spark.kubernetes.driver.master=https://kubernetes.default.svc:6443

How was this patch tested?

run in multi-cluster that do not use the https://kubernetes.default.svc to access the kube-apiserver

wackxu · 2019-12-30T01:14:47Z

/assign @liyinan926

wackxu · 2020-01-09T02:52:10Z

@liyinan926 Could you take a look at this?

wackxu · 2020-01-11T10:59:38Z

@dongjoon-hyun Could you take a look at this?

dongjoon-hyun · 2020-01-11T21:03:36Z

Thank you for pinging me, @wackxu.

dongjoon-hyun · 2020-01-11T21:04:00Z

ok to test

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

SparkQA · 2020-01-11T21:23:08Z

Test build #116537 has finished for PR 27029 at commit 05820de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

One additional request is the following.

Please remove "refer kubeflow/spark-operator#706" from the PR description.

You had better make this PR complete by itself and you can do.

SparkQA · 2020-01-11T21:45:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21323/

SparkQA · 2020-01-11T22:05:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21323/

docs/running-on-kubernetes.md

SparkQA · 2020-01-16T09:12:16Z

Test build #116838 has finished for PR 27029 at commit 1cc2de5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-16T09:48:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21610/

SparkQA · 2020-01-16T10:12:09Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21610/

docs/running-on-kubernetes.md

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

dongjoon-hyun · 2020-01-17T19:37:27Z

Hi, @wackxu Could you address @liyinan926 's comments?

SparkQA · 2020-01-19T09:09:15Z

Test build #117000 has finished for PR 27029 at commit 3c0d369.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-19T09:42:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21767/

SparkQA · 2020-01-19T10:16:38Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21767/

SparkQA · 2020-01-19T11:39:50Z

Test build #117003 has finished for PR 27029 at commit a8cb113.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wackxu · 2020-01-19T11:47:09Z

Hi, @wackxu Could you address @liyinan926 's comments?

@liyinan926 @dongjoon-hyun Sorry for the deplay, updated.

SparkQA · 2020-01-19T12:17:52Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21770/

SparkQA · 2020-01-19T12:48:41Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/21770/

dongjoon-hyun · 2020-01-19T22:11:23Z

The only failure in K8S integration tests is the following, but it's known to be flaky in these days.

- Launcher client dependencies *** FAILED ***

dongjoon-hyun

+1, LGTM. Thank you, @wackxu and @liyinan926 .
Merged to master.

dongjoon-hyun · 2020-01-19T22:17:18Z

@wackxu . You are added to the Apache Spark contributor group. Congratulation!

wackxu · 2020-01-20T01:12:08Z

@dongjoon-hyun @liyinan926 Thank you for your patient response and detailed review.

### What changes were proposed in this pull request? make KUBERNETES_MASTER_INTERNAL_URL configurable ### Why are the changes needed? we do not always use the default port number 443 to access our kube-apiserver, and even in some mulit-tenant cluster, people do not use the service `kubernetes.default.svc` to access the kube-apiserver, so make the internal master configurable is necessary。 ### Does this PR introduce any user-facing change? user can configure the internal master url by ``` --conf spark.kubernetes.internal.master=https://kubernetes.default.svc:6443 ``` ### How was this patch tested? run in multi-cluster that do not use the https://kubernetes.default.svc to access the kube-apiserver Closes apache#27029 from wackxu/internalmaster. Authored-by: xushiwei 00425595 <xushiwei5@huawei.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

wackxu mentioned this pull request Dec 30, 2019

Config port of kubernetes.default.svc from default 443 to 6443 kubeflow/spark-operator#706

Closed

dongjoon-hyun changed the title ~~[SPARK-30371][K8S]make KUBERNETES_MASTER_INTERNAL_URL configurable~~ [SPARK-30371][K8S] Make KUBERNETES_MASTER_INTERNAL_URL configurable Jan 11, 2020

dongjoon-hyun reviewed Jan 11, 2020

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated Show resolved Hide resolved

dongjoon-hyun added the KUBERNETES label Jan 11, 2020

dongjoon-hyun requested changes Jan 11, 2020

View reviewed changes

liyinan926 reviewed Jan 11, 2020

View reviewed changes

docs/running-on-kubernetes.md Outdated Show resolved Hide resolved

wackxu force-pushed the internalmaster branch 2 times, most recently from 9091a3d to 1cc2de5 Compare January 16, 2020 08:58

liyinan926 reviewed Jan 16, 2020

View reviewed changes

docs/running-on-kubernetes.md Outdated Show resolved Hide resolved

liyinan926 reviewed Jan 16, 2020

View reviewed changes

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala Outdated Show resolved Hide resolved

wackxu force-pushed the internalmaster branch from 1cc2de5 to 3c0d369 Compare January 19, 2020 08:56

[SPARK-30371][K8S]make KUBERNETES_MASTER_INTERNAL_URL configurable

a8cb113

wackxu force-pushed the internalmaster branch from 3c0d369 to a8cb113 Compare January 19, 2020 11:21

dongjoon-hyun approved these changes Jan 19, 2020

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-30371][K8S] Make KUBERNETES_MASTER_INTERNAL_URL configurable~~ [SPARK-30371][K8S] Add spark.kubernetes.driver.master conf Jan 19, 2020

dongjoon-hyun closed this in f14061c Jan 19, 2020

wackxu deleted the internalmaster branch January 20, 2020 06:17

ayuan37 mentioned this pull request Feb 22, 2021

[SPARK-30371][K8S] Add spark.kubernetes.driver.master conf palantir/spark#736

Merged

Comments

Conversation

wackxu commented Dec 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

wackxu commented Dec 30, 2019

Uh oh!

wackxu commented Jan 9, 2020

Uh oh!

wackxu commented Jan 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 11, 2020

Uh oh!

dongjoon-hyun commented Jan 11, 2020

Uh oh!

Uh oh!

SparkQA commented Jan 11, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 11, 2020

Uh oh!

SparkQA commented Jan 11, 2020

Uh oh!

Uh oh!

SparkQA commented Jan 16, 2020

Uh oh!

SparkQA commented Jan 16, 2020

Uh oh!

SparkQA commented Jan 16, 2020

Uh oh!

Uh oh!

Uh oh!

dongjoon-hyun commented Jan 17, 2020

Uh oh!

SparkQA commented Jan 19, 2020

Uh oh!

SparkQA commented Jan 19, 2020

Uh oh!

SparkQA commented Jan 19, 2020

Uh oh!

SparkQA commented Jan 19, 2020

Uh oh!

wackxu commented Jan 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jan 19, 2020

Uh oh!

SparkQA commented Jan 19, 2020

Uh oh!

dongjoon-hyun commented Jan 19, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 19, 2020

Uh oh!

wackxu commented Jan 20, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wackxu commented Dec 28, 2019 •

edited

Loading

wackxu commented Jan 11, 2020 •

edited

Loading

wackxu commented Jan 19, 2020 •

edited

Loading