[SPARK-10652][SPARK-10742][Streaming] Set meaningful job descriptions for all streaming jobs #8791

tdas · 2015-09-17T00:22:51Z

Here is the screenshot after adding the job descriptions to threads that run receivers and the scheduler thread running the batch jobs.

All jobs page

Added job descriptions with links to relevant batch details page

All stages page

Added stage descriptions with links to relevant batch details page

Streaming batch details page

Added the +details link

tdas · 2015-09-17T00:23:07Z

@zsxwing Can you take a look at the changes.

zsxwing · 2015-09-17T00:37:17Z

streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala

else is missing.

SparkQA · 2015-09-17T00:40:28Z

Test build #42562 has finished for PR 8791 at commit e8e7894.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-17T03:10:02Z

Test build #42573 has finished for PR 8791 at commit 7206f2e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-09-17T03:32:20Z

@zsxwing I addressed comments and made another update. Now the job description supports having links in them, and that is used to link to the batch details page.

SparkQA · 2015-09-17T03:41:44Z

Test build #42574 has finished for PR 8791 at commit b15012f.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-17T08:54:04Z

Test build #42592 has finished for PR 8791 at commit fd0d435.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-17T09:27:17Z

Test build #42599 has finished for PR 8791 at commit b5cb02e.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-09-17T09:31:54Z

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

Looks it's better to add a new method like

private[streaming] def getStartSite: String = startSite.get()

rather than exposing AtomicReference. In addition, this one won't break MiMa tests.

zsxwing · 2015-09-17T10:15:59Z

LGTM except my comment about startSite.

…tion

SparkQA · 2015-09-18T11:16:54Z

Test build #42659 has finished for PR 8791 at commit d2ab91d.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-09-18T11:27:14Z

This is blocked on #8781 . Once that is merged in both master and 1.5, I will finish this PR by adding unit tests.

tdas · 2015-09-22T01:11:51Z

@andrewor14 Please take a look.

SparkQA · 2015-09-22T03:34:40Z

Test build #42797 has finished for PR 8791 at commit 21c90ea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-22T05:29:53Z

Test build #42815 has finished for PR 8791 at commit 4a45c82.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-22T09:11:07Z

Test build #42825 has finished for PR 8791 at commit 4a0aba8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-22T11:47:12Z

Test build #42826 has finished for PR 8791 at commit f9524c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2015-09-22T15:10:15Z

retest this please

zsxwing · 2015-09-22T16:41:36Z

core/src/main/scala/org/apache/spark/ui/UIUtils.scala

I think both /link and link are called relative links. But /link is called root-relative link. Maybe we should use Links in job descriptions must be root-relative:\n here?

SparkQA · 2015-09-22T17:34:17Z

Test build #42840 has finished for PR 8791 at commit f9524c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-22T21:53:11Z

Test build #42849 has finished for PR 8791 at commit 2fbfd1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-09-22T23:39:34Z

@andrewor14 Could you take a look at this.

andrewor14 · 2015-09-23T00:03:59Z

streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobScheduler.scala

this is assuming the UI is the only consumer of this? I suppose that's OK since no one uses JobLogger anymore

Whoever uses it will get the data as text. Only the UI would actually process it as a link. Nothing breaks.

It looks like this caused a cosmetic issue in the tooltips though: #11845
EDIT: not quite tooltip, but the event timeline display. Also could have been subsequent to this change. Worth a look at the new issue in any event

andrewor14 · 2015-09-23T00:08:03Z

core parts LGTM

zsxwing · 2015-09-23T02:26:00Z

LGTM

tdas · 2015-09-23T02:32:04Z

Thanks @andrewor14 @zsxwing, will merge this to master and 1.5 when tests pass.

SparkQA · 2015-09-23T05:04:07Z

Test build #42878 has finished for PR 8791 at commit 606b284.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ns for all streaming jobs Here is the screenshot after adding the job descriptions to threads that run receivers and the scheduler thread running the batch jobs. ## All jobs page * Added job descriptions with links to relevant batch details page ![image](https://cloud.githubusercontent.com/assets/663212/9924165/cda4a372-5cb1-11e5-91ca-d43a32c699e9.png) ## All stages page * Added stage descriptions with links to relevant batch details page ![image](https://cloud.githubusercontent.com/assets/663212/9923814/2cce266a-5cae-11e5-8a3f-dad84d06c50e.png) ## Streaming batch details page * Added the +details link ![image](https://cloud.githubusercontent.com/assets/663212/9921977/24014a32-5c98-11e5-958e-457b6c38065b.png) Author: Tathagata Das <[email protected]> Closes #8791 from tdas/SPARK-10652. (cherry picked from commit 5548a25) Signed-off-by: Tathagata Das <[email protected]>

Set job descriptions for all streaming jobs

e8e7894

zsxwing reviewed Sep 17, 2015
View reviewed changes

streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala Outdated

Copy link

Member

zsxwing Sep 17, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

else is missing.

tdas added 2 commits September 16, 2015 19:58

Added link to streaming UI in desc of streaming jobs

130e82e

Addressed comments

7206f2e

More fixes

b15012f

Remove println

fd0d435

Use scala.xml's own escaping

b5cb02e

zsxwing reviewed Sep 17, 2015
View reviewed changes

Addressed comments, and added base URL prepending to links in descrip…

d2ab91d

…tion

tdas force-pushed the SPARK-10652 branch from 898765a to d2ab91d Compare September 18, 2015 10:55

tdas added 2 commits September 21, 2015 18:00

Merge remote-tracking branch 'apache-github/master' into SPARK-10652

03bef12

Updated test

21c90ea

tdas changed the title ~~[SPARK-10652][Streaming] Set job descriptions for all streaming jobs~~ [SPARK-10652][Streaming] Set meaniningful job descriptions for all streaming jobs Sep 22, 2015

tdas changed the title ~~[SPARK-10652][Streaming] Set meaniningful job descriptions for all streaming jobs~~ [SPARK-10652][Streaming] Set meaningful job descriptions for all streaming jobs Sep 22, 2015

tdas changed the title ~~[SPARK-10652][Streaming] Set meaningful job descriptions for all streaming jobs~~ [SPARK-10652][SPARK-10742][Streaming] Set meaningful job descriptions for all streaming jobs Sep 22, 2015

Added check for anchors, and unit tests

4a45c82

Add license

4a0aba8

style fix

f9524c6

zsxwing reviewed Sep 22, 2015
View reviewed changes

fixed test

2fbfd1f

andrewor14 reviewed Sep 23, 2015
View reviewed changes

Addressed Andrew's comments

606b284

asfgit closed this in 5548a25 Sep 23, 2015

lw-lin mentioned this pull request Mar 19, 2016

[SPARK-14025][STREAMING][WEBUI] Fix streaming job descriptions on the event timeline #11845

Closed

[SPARK-10652][SPARK-10742][Streaming] Set meaningful job descriptions for all streaming jobs #8791

[SPARK-10652][SPARK-10742][Streaming] Set meaningful job descriptions for all streaming jobs #8791

Uh oh!

Conversation

tdas commented Sep 17, 2015

All jobs page

All stages page

Streaming batch details page

Uh oh!

tdas commented Sep 17, 2015

Uh oh!

zsxwing Sep 17, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 17, 2015

Uh oh!

SparkQA commented Sep 17, 2015

Uh oh!

tdas commented Sep 17, 2015

Uh oh!

SparkQA commented Sep 17, 2015

Uh oh!

SparkQA commented Sep 17, 2015

Uh oh!

SparkQA commented Sep 17, 2015

Uh oh!

zsxwing Sep 17, 2015

Choose a reason for hiding this comment

Uh oh!

zsxwing commented Sep 17, 2015

Uh oh!

SparkQA commented Sep 18, 2015

Uh oh!

tdas commented Sep 18, 2015

Uh oh!

tdas commented Sep 22, 2015

Uh oh!

SparkQA commented Sep 22, 2015

Uh oh!

SparkQA commented Sep 22, 2015

Uh oh!

SparkQA commented Sep 22, 2015

Uh oh!

SparkQA commented Sep 22, 2015

Uh oh!

zsxwing commented Sep 22, 2015

Uh oh!

zsxwing Sep 22, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 22, 2015

Uh oh!

SparkQA commented Sep 22, 2015

Uh oh!

tdas commented Sep 22, 2015

Uh oh!

andrewor14 Sep 23, 2015

Choose a reason for hiding this comment

Uh oh!

tdas Sep 23, 2015

Choose a reason for hiding this comment

Uh oh!

srowen Mar 19, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Sep 23, 2015

Uh oh!

zsxwing commented Sep 23, 2015

Uh oh!

tdas commented Sep 23, 2015

Uh oh!

SparkQA commented Sep 23, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!