[SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI #14461

nblintao · 2016-08-02T16:34:12Z

What changes were proposed in this pull request?

Added a "Back to Master" link on Executor Page of Application Detail UI, if the application is run in standalone mode.

The links link to the main page of Master UI.

How was this patch tested?

Run Spark as a "local cluster" by ./bin/spark-shell --master=local-cluster[2,1,1024].

Then view the Executor Page on Application Detail UI (http://192.168.1.104:4040/executors/). Users could go to the Master Page by the link "Back to Master" on that page.

SparkQA · 2016-08-02T17:00:04Z

Test build #63129 has finished for PR 14461 at commit f35afa2.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class RegisteredApplication(appId: String, master: RpcEndpointRef,

ajbozarth · 2016-08-02T18:00:30Z

I take some time to check out and run this later today, but on a quick pass the code looks ok. I have some issues with the actual pr though. The link on the application summary page is redundant, clicking the spark logo goes back to the master summary already. As for the Executors page, I'll have to test it, but I think there are quite a few situations where a admin might not want a user to access the master UI, even in a standalone cluster.

ajbozarth · 2016-08-05T22:33:49Z

I forgot to comment, but I did check this out and it runs well, I still think the link on the app summary is redundant though.

SparkQA · 2016-08-06T01:49:36Z

Test build #63302 has finished for PR 14461 at commit 0192b37.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

# Conflicts: # project/MimaExcludes.scala

nblintao · 2016-08-06T01:55:18Z

@ajbozarth Thanks for pointing out that. I have removed the link on the application summary page and reverted related changes.

nblintao · 2016-08-06T01:57:43Z

retest it please

SparkQA · 2016-08-06T01:58:55Z

Test build #63303 has finished for PR 14461 at commit 4564bc5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class ShuffleIndexInformation
- public class ShuffleIndexRecord
- case class Least(children: Seq[Expression]) extends Expression
- case class Greatest(children: Seq[Expression]) extends Expression
- case class CreateTable(tableDesc: CatalogTable, mode: SaveMode, query: Option[LogicalPlan])
- case class PreprocessDDL(conf: SQLConf) extends Rule[LogicalPlan]

SparkQA · 2016-08-06T02:33:46Z

Test build #63306 has finished for PR 14461 at commit e3707db.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

nblintao · 2016-08-06T02:46:45Z

[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/project/MimaExcludes.scala:50: ')' expected but '.' found.
[error]       ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.scheduler.SparkListenerApplicationStart.apply")
[error]                     ^
[error] /home/jenkins/workspace/SparkPullRequestBuilder@2/project/MimaExcludes.scala:51: ';' expected but '.' found.
[error]       ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.scheduler.SparkListenerApplicationStart.copy")
[error]                     ^

I think I have fixed the conflict correctly. Why's that?

ajbozarth · 2016-08-06T02:50:08Z

The excludes are a list, you forgot to add commas between them

nblintao · 2016-08-06T02:52:17Z

Oh yes. Thanks :)

SparkQA · 2016-08-06T04:51:13Z

Test build #63308 has finished for PR 14461 at commit 76e68eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-10-13T21:59:35Z

@nblintao I'd like to take another look at this, mind rebasing?

nblintao · 2016-12-29T02:55:02Z

@ajbozarth I finally have a chance to rebase it in the winter break. Could you please have a look? Thanks!

SparkQA · 2016-12-29T05:11:09Z

Test build #70698 has finished for PR 14461 at commit d598e55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2017-01-03T18:37:53Z

Thanks for rebasing, I'll check this out sometime this week

ajbozarth · 2017-01-18T00:05:04Z

@nblintao did you try this code on different cluster types? Like local, standalone, yarn, mesos?

ajbozarth · 2017-01-18T00:05:30Z

Also @srowen and @vanzin could you take a look at this one too?

ajbozarth · 2017-01-19T01:32:37Z

I did some testing and this properly shows up on local and standalone and doesn't show the link on yarn. So this LGTM

vanzin · 2017-01-25T22:47:04Z

core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala

    appAttemptId: Option[String],
-    driverLogs: Option[Map[String, String]] = None) extends SparkListenerEvent
+    driverLogs: Option[Map[String, String]] = None,
+    masterWebUiUrl: Option[String] = None) extends SparkListenerEvent


Do you need this? It seems the standalone registration messages already take care of this.

If you want this, are you intentionally not saving this in event logs? The way it is, this is only ever useful when the application is starting but hasn't yet registered with the master. That's a very short period of time and probably not worth it.

If you add code to save to event logs, I'd rename this to be more cluster manager-agnostic. e.g. "clusterWebUiUrl".

vanzin · 2017-01-25T22:47:32Z

core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala

  private val listener = parent.listener

  def render(request: HttpServletRequest): Seq[Node] = {
+    val masterWebUiUrl = listener.masterWebUiUrl


I think this information makes way more sense in the "Environment" tab.

HyukjinKwon · 2017-02-09T13:56:26Z

(gentle ping @nblintao)

nblintao · 2017-02-09T18:39:26Z

@HyukjinKwon Thanks for reminding. I will have a look asap. Too many things to do recently.

HyukjinKwon · 2017-05-11T13:01:39Z

(gentle ping @nblintao)

nblintao · 2017-05-11T15:46:25Z

@HyukjinKwon Thanks for reminding. I am finals this week. I will fix this after 15th.

# What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 Closes apache#11459 Closes apache#13833 Closes apache#13720 Closes apache#12506 Closes apache#12456 Closes apache#12252 Closes apache#17689 Closes apache#17791 Closes apache#18163 Closes apache#17640 Closes apache#17926 Closes apache#18163 Closes apache#12506 Closes apache#18044 Closes apache#14036 Closes apache#15831 Closes apache#14461 Closes apache#17638 Closes apache#18222 Added: Closes apache#18045 Closes apache#18061 Closes apache#18010 Closes apache#18041 Closes apache#18124 Closes apache#18130 Closes apache#12217 Added: Closes apache#16291 Closes apache#17480 Closes apache#14995 Added: Closes apache#12835 Closes apache#17141 ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18223 from HyukjinKwon/close-stale-prs.

link application page to master page

f35afa2

nblintao changed the title ~~[SPARK-16856] [WEBUI] [CORE] Link application summary page and detail page to the master page~~ [SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI Aug 6, 2016

nblintao added 2 commits August 6, 2016 09:41

Remove the link on the application summary page

1f56f46

Update MimaExcludes

0192b37

Merge branch 'master' into linkMasterPage

4564bc5

# Conflicts: # project/MimaExcludes.scala

remove blanks

e3707db

update MimaExcludes

76e68eb

nblintao added 2 commits December 28, 2016 20:09

Merge remote-tracking branch 'apache/master'

8743898

resolve conflicts

d598e55

ajbozarth mentioned this pull request Jan 18, 2017

[SPARK-16520] [WEBUI] Link executors to corresponding worker pages #14204

Closed

vanzin reviewed Jan 25, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Jun 7, 2017

[INFRA] Close stale PRs #18223

Closed

asfgit closed this in b771fed Jun 8, 2017

[SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI #14461

[SPARK-16856] [WEBUI] [CORE] Link the application's executor page to the master's UI #14461

Uh oh!

Conversation

nblintao commented Aug 2, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Aug 2, 2016

Uh oh!

ajbozarth commented Aug 2, 2016

Uh oh!

ajbozarth commented Aug 5, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

ajbozarth commented Aug 6, 2016

Uh oh!

nblintao commented Aug 6, 2016

Uh oh!

SparkQA commented Aug 6, 2016

Uh oh!

ajbozarth commented Oct 13, 2016

Uh oh!

nblintao commented Dec 29, 2016

Uh oh!

SparkQA commented Dec 29, 2016

Uh oh!

ajbozarth commented Jan 3, 2017

Uh oh!

ajbozarth commented Jan 18, 2017

Uh oh!

ajbozarth commented Jan 18, 2017

Uh oh!

ajbozarth commented Jan 19, 2017

Uh oh!

vanzin Jan 25, 2017

Choose a reason for hiding this comment

Uh oh!

vanzin Jan 25, 2017

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Feb 9, 2017

Uh oh!

nblintao commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented May 11, 2017

Uh oh!

nblintao commented May 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nblintao commented Aug 2, 2016 •

edited

Loading

nblintao commented Feb 9, 2017 •

edited

Loading