Skip to content

[SPARK-36566][K8S] Add Spark appname as a label to pods #34460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Yikun wants to merge 4 commits into from
Closed

[SPARK-36566][K8S] Add Spark appname as a label to pods #34460

Yikun wants to merge 4 commits into from

Conversation

@Yikun
Copy link
Member

@Yikun Yikun commented Nov 1, 2021 •
edited
Loading

What changes were proposed in this pull request?

Add Spark appname as a label to pods.

Note that:

  • there were a SPARK_APP_ID_LABEL: is the unique spark APP applicationId with spark prefix (like "spark-{applicationId}")
  • SPARK_APP_NAME_LABEL in this patch is the Spark APP name, it's more friendly and readable for k8s cluster maintainer to figure out the spark app name of specific pods.

Why are the changes needed?

Then we can find out all pods (driver/executor) list by using: k get pods -l spark.app.name=xxx, also can figure out the spark app name of specific pods.

Does this PR introduce any user-facing change?

Add label to pods.

How was this patch tested?

Add UT

@Yikun
Copy link
Member Author

Yikun commented Nov 1, 2021

cc who marked SPARK-36566: @holdenk @HyukjinKwon

@SparkQA
Copy link

SparkQA commented Nov 1, 2021

Test build #144813 has finished for PR 34460 at commit ca0ecce.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 1, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49283/

@SparkQA
Copy link

SparkQA commented Nov 1, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49283/

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Nov 1, 2021

Thank you, @Yikun .

dongjoon-hyun
dongjoon-hyun reviewed Nov 1, 2021
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had better make the PR and commit log independent as much as possible. Could you elaborate a little more about the difference between spark-app-selector and spark-app-name? It will become a helpful commit log for the other community members.

@Yikun
Copy link
Member Author

Yikun commented Nov 2, 2021

Could you elaborate a little more about the difference between spark-app-selector and spark-app-name?

Sure, done!

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Test build #144828 has finished for PR 34460 at commit e7e127a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49298/

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49298/

@Yikun
Copy link
Member Author

Yikun commented Nov 2, 2021

Need to address invalid label name...

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Test build #144840 has finished for PR 34460 at commit 69fe632.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49310/

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49310/

@Yikun
Copy link
Member Author

Yikun commented Nov 2, 2021

Label length limit: contain at most 63 characters, pod name limit: contain at most 253 characters

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Test build #144848 has finished for PR 34460 at commit 61c462c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49318/

@SparkQA
Copy link

SparkQA commented Nov 2, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49318/

@Yikun
Copy link
Member Author

Yikun commented Nov 3, 2021

@dongjoon-hyun Looks like failure is unrelated, could you take a look again? Thanks!

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Nov 3, 2021

Sure, thank you for updating, @Yikun .

dongjoon-hyun
dongjoon-hyun approved these changes Nov 3, 2021
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @Yikun .
Merged to master for Apache Spark 3.3.

@Yikun Yikun mentioned this pull request Nov 18, 2021
Closed
@turboFei turboFei mentioned this pull request Apr 16, 2025
Closed
turboFei added a commit to apache/kyuubi that referenced this pull request Apr 17, 2025
@turboFei
[KYUUBI #7034] [KUBERNETES] Prefer to use pod spark-app-name label …
cc68cb4 
…as application name than pod name

### Why are the changes needed?

After apache/spark#34460 (Since Spark 3.3.0), the `spark-app-name` is available.

We shall use it as the application name if it exists.

### How was this patch tested?

Minor change.
### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7034 from turboFei/k8s_app_name.

Closes #7034

bfa88a4 [Wang, Fei] Get pod app name

Authored-by: Wang, Fei <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
turboFei added a commit to apache/kyuubi that referenced this pull request Apr 17, 2025
@turboFei
[KYUUBI #7034] [KUBERNETES] Prefer to use pod spark-app-name label …
a326ba7 
…as application name than pod name

### Why are the changes needed?

After apache/spark#34460 (Since Spark 3.3.0), the `spark-app-name` is available.

We shall use it as the application name if it exists.

### How was this patch tested?

Minor change.
### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #7034 from turboFei/k8s_app_name.

Closes #7034

bfa88a4 [Wang, Fei] Get pod app name

Authored-by: Wang, Fei <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
(cherry picked from commit cc68cb4)
Signed-off-by: Wang, Fei <[email protected]>
turboFei added a commit to turboFei/kyuubi that referenced this pull request Aug 27, 2025
@turboFei
[KYUUBI apache#7034] [KUBERNETES] Prefer to use pod spark-app-name …
3048add 
…label as application name than pod name

### Why are the changes needed?

After apache/spark#34460 (Since Spark 3.3.0), the `spark-app-name` is available.

We shall use it as the application name if it exists.

### How was this patch tested?

Minor change.
### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#7034 from turboFei/k8s_app_name.

Closes apache#7034

bfa88a4 [Wang, Fei] Get pod app name

Authored-by: Wang, Fei <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

KUBERNETES

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Yikun @SparkQA @dongjoon-hyun