[SPARK-2458] Make failed application log visible on History Server #3467

tsudukim · 2014-11-26T03:23:40Z

Enabled HistoryServer to show incomplete applications.
We can see the log for incomplete applications by clicking the bottom link.

tsudukim · 2014-11-26T03:24:19Z

In this PR, the sort order is (-endTime, -startTime) which means that
the sorting is still end time for completed apps but is start time for incomplete apps
because incomplete apps don't have the end time generally.

Please see also the discuss in this previous PR about the need of this feature. #1558

AmplabJenkins · 2014-11-26T03:27:11Z

Can one of the admins verify this patch?

tsudukim · 2014-11-26T03:27:33Z

And these are the screenshots of new UI.
A new link for the page of incomplete applications is added at bottom. ("Show incomplete applications")

And this is the new separated page for incomplete applications.
Incomplete apps donsn't have the end time generally so end time are all shown as "-".

andrewor14 · 2014-11-26T04:26:31Z

Looks great. I will take a detailed pass once the 1.2 release is out of the window.

andrewor14 · 2014-11-26T04:26:39Z

add to whitelist

SparkQA · 2014-11-26T04:30:14Z

Test build #23875 has started for PR 3467 at commit 66fc15b.

This patch merges cleanly.

SparkQA · 2014-11-26T04:31:23Z

Test build #23875 has finished for PR 3467 at commit 66fc15b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-26T04:31:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23875/
Test FAILed.

SparkQA · 2014-11-26T13:10:09Z

Test build #23892 has started for PR 3467 at commit e8cdf7b.

This patch merges cleanly.

SparkQA · 2014-11-26T14:30:23Z

Test build #23892 has finished for PR 3467 at commit e8cdf7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-26T14:30:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23892/
Test PASSed.

ryan-williams · 2014-11-28T14:34:18Z

core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala

requestedIncomplete == app.incomplete might be simpler

tsudukim · 2014-11-28T19:11:00Z

@ryan-williams Thank you for your review!
I fixed them.

SparkQA · 2014-11-28T19:15:16Z

Test build #23941 has started for PR 3467 at commit d6dd750.

This patch merges cleanly.

SparkQA · 2014-11-28T20:37:12Z

Test build #23941 has finished for PR 3467 at commit d6dd750.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-28T20:37:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23941/
Test PASSed.

andrewor14 · 2014-12-10T20:35:08Z

Hey @tsudukim you will need to rebase to master because there have been changes to the HS that went in recently. Once you do that I will take a detailed pass through the PR.

SparkQA · 2014-12-12T18:02:56Z

Test build #24413 has started for PR 3467 at commit 5454869.

This patch merges cleanly.

tsudukim · 2014-12-12T18:08:24Z

@andrewor14 Thank you for following this ticket.
I finished rebasing to master.

SparkQA · 2014-12-12T19:19:22Z

Test build #24413 has finished for PR 3467 at commit 5454869.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-12T19:19:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24413/
Test FAILed.

andrewor14 · 2014-12-18T19:49:03Z

retest this please

SparkQA · 2014-12-18T19:52:44Z

Test build #24596 has started for PR 3467 at commit 5454869.

This patch merges cleanly.

SparkQA · 2014-12-18T21:12:30Z

Test build #24596 has finished for PR 3467 at commit 5454869.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class RandomModuleHook(object):
- class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)

AmplabJenkins · 2014-12-18T21:12:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24596/
Test PASSed.

andrewor14 · 2014-12-18T21:27:19Z

By the way I'm going to block this on #1222, since that one has been open for a while and I would like to get it merged first before @vanzin has to resolve more conflicts again. After that one goes in this PR will likely have to resolve non-trivial merge conflicts. I might hold back reviewing this PR until then. Just a heads up.

andrewor14 · 2014-12-20T02:42:08Z

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

what does this mean? Which column does it sort by?

About completed applications, they are sorted by endTime because they have proper endTime (almost unique).
About incomplete applications, they are sorted by startTime because they have the same invalid endTime(-1). As the first order is not effective, the second order is effective.

andrewor14 · 2014-12-20T02:42:25Z

core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala

space before if

also might be good to check for info.endTime > 0 instead, here and other places

andrewor14 · 2014-12-20T02:47:51Z

Hey @tsudukim this patch is pretty straightforward and it looks good. Unfortunately you will need to resolve a non-trivial merge conflict with another patch that I just merged.

tsudukim · 2014-12-24T10:29:52Z

Thank you for your comments! I'm going to do it in a few days!

Enabled HistoryServer to show incomplete applications. We can see the log for incomplete applications by clicking the bottom link.

Fixed "Incomplete" as capitalized in the web UI. Modified double negative variable name.

SparkQA · 2014-12-25T09:37:31Z

Test build #24818 has started for PR 3467 at commit f9ef854.

This patch merges cleanly.

SparkQA · 2014-12-25T10:41:31Z

Test build #24818 has finished for PR 3467 at commit f9ef854.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-25T10:41:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24818/
Test FAILed.

…-2458-2

SparkQA · 2014-12-26T04:22:32Z

Test build #24830 has started for PR 3467 at commit 76205d2.

This patch merges cleanly.

SparkQA · 2014-12-26T05:44:38Z

Test build #24830 has finished for PR 3467 at commit 76205d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-26T05:44:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24830/
Test PASSed.

tsudukim · 2015-01-06T08:14:40Z

resolved conflicts!

andrewor14 · 2015-01-07T15:29:58Z

core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala

Return true when the application has completed.

Don't worry I'll fix the wording myself when I merge this

andrewor14 · 2015-01-07T15:31:39Z

Alright LGTM I'm merging this into master thanks!

tsudukim mentioned this pull request Nov 26, 2014

[SPARK-2458] Make failed application log visible on History Server #1558

Closed

ryan-williams reviewed Nov 28, 2014
View reviewed changes

tsudukim force-pushed the feature/SPARK-2458-2 branch from e69329b to 5454869 Compare December 12, 2014 18:01

andrewor14 reviewed Dec 20, 2014
View reviewed changes

tsudukim added 4 commits December 25, 2014 17:48

[SPARK-2458] Make failed application log visible on History Server

08ea14d

Enabled HistoryServer to show incomplete applications. We can see the log for incomplete applications by clicking the bottom link.

Modified too long lines.

3ed8a41

Modified typo and better implementation.

9b465b0

Added space between "if" and "(".

f9ef854

Fixed "Incomplete" as capitalized in the web UI. Modified double negative variable name.

tsudukim force-pushed the feature/SPARK-2458-2 branch from 262dc52 to f9ef854 Compare December 25, 2014 09:35

tsudukim added 2 commits December 26, 2014 10:47

Merge branch 'master' of github.com:tsudukim/spark into feature/SPARK…

29a04a9

…-2458-2

Fixed and added test code.

76205d2

andrewor14 reviewed Jan 7, 2015
View reviewed changes

asfgit closed this in 6e74ede Jan 7, 2015

XuTingjun mentioned this pull request Jun 2, 2015

[SPARK-7889] [UI] make sure click the "App ID" on HistoryPage, the SparkUI will be refreshed. #6545

Closed

[SPARK-2458] Make failed application log visible on History Server #3467

[SPARK-2458] Make failed application log visible on History Server #3467

Uh oh!

Conversation

tsudukim commented Nov 26, 2014

Uh oh!

tsudukim commented Nov 26, 2014

Uh oh!

AmplabJenkins commented Nov 26, 2014

Uh oh!

tsudukim commented Nov 26, 2014

Uh oh!

andrewor14 commented Nov 26, 2014

Uh oh!

andrewor14 commented Nov 26, 2014

Uh oh!

SparkQA commented Nov 26, 2014

Uh oh!

SparkQA commented Nov 26, 2014

Uh oh!

AmplabJenkins commented Nov 26, 2014

Uh oh!

SparkQA commented Nov 26, 2014

Uh oh!

SparkQA commented Nov 26, 2014

Uh oh!

AmplabJenkins commented Nov 26, 2014

Uh oh!

ryan-williams Nov 28, 2014

Choose a reason for hiding this comment

Uh oh!

tsudukim commented Nov 28, 2014

Uh oh!

SparkQA commented Nov 28, 2014

Uh oh!

SparkQA commented Nov 28, 2014

Uh oh!

AmplabJenkins commented Nov 28, 2014

Uh oh!

andrewor14 commented Dec 10, 2014

Uh oh!

SparkQA commented Dec 12, 2014

Uh oh!

tsudukim commented Dec 12, 2014

Uh oh!

SparkQA commented Dec 12, 2014

Uh oh!

AmplabJenkins commented Dec 12, 2014

Uh oh!

andrewor14 commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

SparkQA commented Dec 18, 2014

Uh oh!

AmplabJenkins commented Dec 18, 2014

Uh oh!

andrewor14 commented Dec 18, 2014

Uh oh!

andrewor14 Dec 20, 2014

Choose a reason for hiding this comment

Uh oh!

tsudukim Dec 25, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 Jan 7, 2015

Choose a reason for hiding this comment

Uh oh!

andrewor14 Dec 20, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 Dec 20, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Dec 20, 2014

Uh oh!

tsudukim commented Dec 24, 2014

Uh oh!

SparkQA commented Dec 25, 2014

Uh oh!