[SPARK-8171] [Web UI] Javascript based infinite scrolling for the log page #10910

ajbozarth · 2016-01-26T00:58:26Z

Updated the log page by replacing the current pagination with a javascript-based infinite scroll solution

ajbozarth · 2016-01-26T00:58:38Z

I spent some time looking into ways to use javascript to load the content rather than use a page refresh and all the solutions I found required that the file has to be available on the web-server itself, which is very difficult to do with a continuously updating log file.

Instead I implemented a alteration to the current pagination that adds to the top (or bottom) of the log a page at a time rather than limiting you to read only a page at a time. By setting the scroll starting location at page load this simulates an infinite scroll even though the page is re-loading.

I also set the log to scroll to the bottom on initial page load.

Given this is inspired by but not exactly the original jira and does distinctly change the use of this page I would love to get peoples opinions on it's usability, look and feel, and if it's worth the change at all.

JoshRosen · 2016-01-26T01:02:28Z

Screenshot? Please include them for all UI changes.

ajbozarth · 2016-01-26T01:10:25Z

Sorry took me a bit longer to get them ready than usual.

Initial page load:

initial load scrolled to top:

after "load more" is clicked and page reloads:

if the full file is loaded:

when you scroll to the bottom after loading more and there is new log available:

SparkQA · 2016-01-26T02:58:45Z

Test build #50050 has finished for PR 10910 at commit 4d7c433.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-01-26T03:21:35Z

Retest this please

SparkQA · 2016-01-26T05:21:48Z

Test build #50065 has finished for PR 10910 at commit 4d7c433.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2016-01-26T23:20:41Z

So every time the user clicks "load more" it will request/fetch all of the log that has already been loaded again? (Just trying to understand the design).

ajbozarth · 2016-01-26T23:32:36Z

When the user clicks load more the page refreshes and on the new page load it reads a larger chunk of the file.
The main implementation difference between the previous version and this is that when the user requests another page of the log it is added to the current log page(s) rather than just showing the new requested page alone.

JoshRosen · 2016-01-26T23:36:22Z

This isn't quite what I had in mind when filing that JIRA; the UX that I envisioned was something where if a user clicks "load more" the Javascript requests only the new / additional portion log output from the backend then update the UI without reloading the page.

ajbozarth · 2016-01-26T23:42:54Z

I had that in mind too, I just had this implementation in my head as an alternative and wanted to see what you thought. If you'd rather stick with what we have for now and wait until we figure out a js solution, then we can close this PR. I'll also spend just a bit more time trying to find a js solution to fetching the file data before moving to another task.

JoshRosen · 2016-01-27T01:38:40Z

Yeah, let's close this for the time being and re-open when you have a new approach which is ready to be reviewed.

ajbozarth · 2016-01-30T07:27:09Z

Thanks @JoshRosen for rejecting my first version, making it did helped me find this new solution using js though.

Few questions:
Should we load the content of the log-data span and/or the logText content dynamically on initial page load or leave it hard-coded as is?
Would we rather have a proper infinite scroll where it detects when the user reaches the top or bottom rather than the buttons I've implemented? I prefer the buttons myself.

ajbozarth · 2016-01-30T07:28:17Z

I'll post some new screen shots below but the look and feel is basically the same as the previous fake version.
I've changed the button text a bit and added an transient info alert when the user tries to load new and nothing is available. Note it doesn't replace the button but appears above it.

ajbozarth · 2016-01-30T07:33:36Z

@ckadner this was what I was showing you if you want to take a look

SparkQA · 2016-01-30T09:36:39Z

Test build #50439 has finished for PR 10910 at commit 35e08c7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2016-02-01T21:57:00Z

@sarutak

ajbozarth · 2016-02-03T01:55:23Z

@JoshRosen opinions on the updated version?

ajbozarth · 2016-02-08T18:58:32Z

Just in case it fell through the cracks last week, what do people think of the new javascript-based log page?

ajbozarth · 2016-02-11T16:06:19Z

@tgravescs thanks for all the help on those other PRs, mind taking a look at this one?

ajbozarth · 2016-02-25T22:43:54Z

@sarutak @JoshRosen @tgravescs in case this slipped past you guys due to Spark Summit last week

ajbozarth · 2016-03-21T16:33:12Z

@JoshRosen If you have time to take a second look at this, it's been sitting in limbo for a while, thanks

ajbozarth · 2016-03-28T21:54:00Z

@sarutak @JoshRosen @tgravescs I know you guys are busy getting ready for Spark Summit West, but I was wondering if you had an eta on looking at this? It's been open for a couple months now, thanks.

sarutak · 2016-03-29T01:46:40Z

@ajbozarth I'll check it soon.

sarutak · 2016-04-04T04:29:43Z

core/src/main/resources/org/apache/spark/ui/static/log-view.js

+
+function tailLog() {
+  $(".log-content").scrollTop($(".log-content")[0].scrollHeight);
+}


It's not good to repeat a query for the same dom-objects like $(.log-content).
Similar repetitions are in some places in this change.

sarutak · 2016-04-04T11:52:38Z

@ajbozarth Sorry for having you wait. I've inspected quickly and left some comments.

@JoshRosen I think this feature itself is useful but is this the feature you had in mind?

ajbozarth · 2016-04-04T22:08:08Z

@sarutak thanks for the feedback, I'll look through and address them as I have time over the next couple days

ajbozarth · 2016-04-07T20:22:41Z

@sarutak I just went through your comments and responded and made fixes. I'm currently testing locally and will push my changes later today.

SparkQA · 2016-04-08T00:10:53Z

Test build #55251 has finished for PR 10910 at commit 0f49738.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajbozarth · 2016-04-13T18:57:57Z

@sarutak @JoshRosen What do you think of the latest changes?

sarutak · 2016-04-15T04:01:20Z

@ajbozarth Thank you for the update. I'll check it next Sunday or Monday.

sarutak · 2016-04-18T07:10:09Z

core/src/main/resources/org/apache/spark/ui/static/log-view.js

+    url: "/log" + baseParams + "&offset=" + offset + "&byteLength=" + moreByteLength,
+    success: function (data) {
+      var oldHeight = $(".log-content")[0].scrollHeight;
+      var newlineIndex = data.indexOf('\n')


nit: semicolon.

sarutak · 2016-04-18T07:13:01Z

@ajbozarth I've left some comments about very trivial nits, otherwise LGTM.

sarutak · 2016-04-18T07:14:00Z

core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala

-          Previous 0 B
-        </button>
-      }
+    val curLogLength = endByte - startByte;


nit:semicolon is not needed here.

ajbozarth · 2016-04-18T15:06:43Z

Fixed the semicolons, switching back and forth between js and scala when they're "optional" on both really messes with you. Thanks @sarutak

SparkQA · 2016-04-18T20:07:53Z

Test build #56098 has finished for PR 10910 at commit 293f1a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2016-04-18T23:36:42Z

LGTM. If we have no other comments within 24 hours, I'll merge this.

sarutak · 2016-04-20T12:22:40Z

Merging this into master. Thanks @ajbozarth !

simulated infinite scrolling using current pagination code

4d7c433

ajbozarth closed this Jan 27, 2016

ajbozarth added 3 commits January 27, 2016 12:57

Merge branch 'master' into spark8171

ab20345

updated log page to use javascript for infinite scrolling

f37fa32

Merge branch 'master' into spark8171

35e08c7

ajbozarth reopened this Jan 30, 2016

ajbozarth changed the title ~~[SPARK-8171] [Web UI] Simulated infinite scrolling using current pagination code in Spark log viewers~~ [SPARK-8171] [Web UI] Javascript based infinite scrolling for the log page Feb 26, 2016

sarutak reviewed Apr 4, 2016
View reviewed changes

Merge branch 'master' into spark8171

03d16aa

Addressed js comments

0f49738

sarutak reviewed Apr 18, 2016
View reviewed changes

fixed some semicolons

293f1a4

asfgit closed this in 8342778 Apr 20, 2016

ajbozarth deleted the spark8171 branch April 20, 2016 16:09

[SPARK-8171] [Web UI] Javascript based infinite scrolling for the log page #10910

[SPARK-8171] [Web UI] Javascript based infinite scrolling for the log page #10910

Uh oh!

Conversation

ajbozarth commented Jan 26, 2016

Uh oh!

ajbozarth commented Jan 26, 2016

Uh oh!

JoshRosen commented Jan 26, 2016

Uh oh!

ajbozarth commented Jan 26, 2016

Uh oh!

SparkQA commented Jan 26, 2016

Uh oh!

ajbozarth commented Jan 26, 2016

Uh oh!

SparkQA commented Jan 26, 2016

Uh oh!

holdenk commented Jan 26, 2016

Uh oh!

ajbozarth commented Jan 26, 2016

Uh oh!

JoshRosen commented Jan 26, 2016

Uh oh!

ajbozarth commented Jan 26, 2016

Uh oh!

JoshRosen commented Jan 27, 2016

Uh oh!

ajbozarth commented Jan 30, 2016

Uh oh!

ajbozarth commented Jan 30, 2016

Uh oh!

ajbozarth commented Jan 30, 2016

Uh oh!

SparkQA commented Jan 30, 2016

Uh oh!

andrewor14 commented Feb 1, 2016

Uh oh!

ajbozarth commented Feb 3, 2016

Uh oh!

ajbozarth commented Feb 8, 2016

Uh oh!

ajbozarth commented Feb 11, 2016

Uh oh!

ajbozarth commented Feb 25, 2016

Uh oh!

ajbozarth commented Mar 21, 2016

Uh oh!

ajbozarth commented Mar 28, 2016

Uh oh!

sarutak commented Mar 29, 2016

Uh oh!

sarutak Apr 4, 2016

Choose a reason for hiding this comment

Uh oh!

sarutak commented Apr 4, 2016

Uh oh!

ajbozarth commented Apr 4, 2016

Uh oh!

ajbozarth commented Apr 7, 2016

Uh oh!

SparkQA commented Apr 8, 2016

Uh oh!

ajbozarth commented Apr 13, 2016

Uh oh!

sarutak commented Apr 15, 2016

Uh oh!

sarutak Apr 18, 2016

Choose a reason for hiding this comment

Uh oh!

sarutak commented Apr 18, 2016

Uh oh!

sarutak Apr 18, 2016

Choose a reason for hiding this comment

Uh oh!

ajbozarth commented Apr 18, 2016

Uh oh!

SparkQA commented Apr 18, 2016

Uh oh!

sarutak commented Apr 18, 2016