[SPARK-31642] Add Pagination Support for Structured Streaming Page #28485

iRakson · 2020-05-09T12:49:16Z

What changes were proposed in this pull request?

Add Pagination Support for structured streaming page. Now both tables Active Queries and Completed Queries will have pagination.
To implement pagination, pagination framework from #7399 is used.

Also tables will only be shown if there is at least one entry in the table.

Why are the changes needed?

This will help users in analysing their structured streaming queries in much better way.
Other Web UI pages support pagination in their table. So this will make web UI more consistent across pages.
This can prevent potential OOM errors.

Does this PR introduce any user-facing change?

Yes. Both tables will support pagination.

How was this patch tested?

Before Changes:

After Changes:

iRakson · 2020-05-09T12:50:23Z

cc @srowen @zsxwing @sarutak @uncleGen Kindly Review

HyukjinKwon · 2020-05-10T04:05:03Z

ok to test

HyukjinKwon · 2020-05-10T04:05:10Z

cc @gengliangwang and @HeartSaVioR as well

SparkQA · 2020-05-10T07:05:01Z

Test build #122469 has finished for PR 28485 at commit df61b7c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class StreamingQueryPagedTable(
case class StructuredStreamingRow(
class StreamingQueryDataSource(uiData: Seq[StreamingQueryUIData], sortColumn: String, desc: Boolean,

iRakson · 2020-05-10T07:29:43Z

retest this please

srowen · 2020-05-10T17:59:02Z

Jenkins retest this please

SparkQA · 2020-05-10T21:43:19Z

Test build #122479 has finished for PR 28485 at commit df61b7c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class StreamingQueryPagedTable(
case class StructuredStreamingRow(
class StreamingQueryDataSource(uiData: Seq[StreamingQueryUIData], sortColumn: String, desc: Boolean,

uncleGen · 2020-05-11T01:59:32Z

Seems like an unrelated and flaky failure and will fixed in #28391

uncleGen · 2020-05-11T02:00:30Z

Jenkins retest this please

SparkQA · 2020-05-11T06:25:28Z

Test build #122485 has finished for PR 28485 at commit df61b7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class StreamingQueryPagedTable(
case class StructuredStreamingRow(
class StreamingQueryDataSource(uiData: Seq[StreamingQueryUIData], sortColumn: String, desc: Boolean,

uncleGen

Bunch of codes seem to be copied from codebase, it might be better to find the way how to reuse them.

uncleGen · 2020-05-11T06:07:14Z

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala

+    s"&$pageNumberFormField=$page" +
+    s"&$tableTag.sort=$encodedSortColumn" +
+    s"&$tableTag.desc=$sortDesc" +
+    s"&$pageSizeFormField=$pageSize" +
+    s"#$tableTag"


nit: indent

iRakson · 2020-05-11T06:42:22Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/UISeleniumSuite.scala

          }

-          eventually(timeout(30.seconds), interval(100.milliseconds)) {
+          eventually(timeout(300.seconds), interval(100.milliseconds)) {


This was modified accidentally, I will update this.

SparkQA · 2020-05-11T12:22:14Z

Test build #122493 has finished for PR 28485 at commit 2486177.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-12T14:21:34Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/UISeleniumSuite.scala

-              List("Name", "Status", "Id", "Run ID", "Start Time", "Duration", "Avg Input /sec",
-                "Avg Process /sec", "Lastest Batch")
+
+            val arrow = 0x25BE.toChar


nit: Why not just simply use a literal like '\u25BE' ?

uncleGen · 2020-05-13T02:37:11Z

LGTM. Pending on #28512 accepted and then update this PR.

### What changes were proposed in this pull request? Currently while implementing pagination using the existing pagination framework, a lot of code is being copied as pointed out [here](#28485 (review)). I introduced some changes in `PagedTable` which is the main trait for implementing the pagination. * Added function for getting table parameters. * Added a function for table header row. This will help in maintaining consistency across the tables. All the header rows across tables will be consistent now. ### Why are the changes needed? * A lot of code is copied every time pagination is implemented for any table. * Code readability is not great as lot of HTML is embedded. * Paginating other tables will be a lot easier now. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually. This is mainly refactoring work, no new functionality introduced. Existing test cases should pass. Closes #28512 from iRakson/refactorPaginationFramework. Authored-by: iRakson <[email protected]> Signed-off-by: Sean Owen <[email protected]>

iRakson · 2020-05-21T19:13:22Z

@srowen @sarutak @uncleGen This PR has been updated.

SparkQA · 2020-05-22T00:33:53Z

Test build #122941 has finished for PR 28485 at commit 088c551.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-05-22T15:46:50Z

Jenkins retest this please

SparkQA · 2020-05-22T20:53:13Z

Test build #123008 has finished for PR 28485 at commit 088c551.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-05-23T08:19:39Z

I've merged to master.
Thanks @iRakson !

iRakson · 2020-05-23T09:03:57Z

Thank You 😊 @sarutak @srowen @uncleGen

iRakson · 2020-05-24T14:58:13Z

sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala

+        ("Run ID", true, None),
+        ("Start Time", true, None),
+        ("Duration", true, None),
+        ("Avg Input /sec", true, None),


@srowen @uncleGen @sarutak Should we add tooltips for these columns.

gatorsmile · 2020-05-27T02:54:18Z

. I will add snapshots soon.

@iRakson could you update the PR description with the screenshot.

iRakson · 2020-05-27T18:57:07Z

@iRakson could you update the PR description with the screenshot.

Updated.

…us column sortable ### What changes were proposed in this pull request? In #28485 pagination support for tables of Structured Streaming Tab was added. It missed 2 things: * For sorting duration column, `String` was used which sometimes gives wrong results(consider `"3 ms"` and `"12 ms"`). Now we first sort the duration column and then convert it to readable String * Status column was not made sortable. ### Why are the changes needed? To fix the wrong result for sorting and making Status column sortable. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? After changes: <img width="1677" alt="Screenshot 2020-06-08 at 2 18 48 PM" src="https://user-images.githubusercontent.com/15366835/84010992-153fa280-a993-11ea-9846-bf176f2ec5d7.png"> Closes #28752 from iRakson/ssTests. Authored-by: iRakson <[email protected]> Signed-off-by: Sean Owen <[email protected]>

probot-autolabeler bot added SQL STRUCTURED STREAMING WEB UI labels May 9, 2020

uncleGen reviewed May 11, 2020

View reviewed changes

iRakson commented May 11, 2020

View reviewed changes

iRakson mentioned this pull request May 11, 2020

[SPARK-31638][WEBUI]Clean Pagination code for all webUI pages #28448

Closed

srowen approved these changes May 12, 2020

View reviewed changes

sarutak reviewed May 12, 2020

View reviewed changes

iRakson mentioned this pull request May 12, 2020

[SPARK-31688][WEBUI] Refactor Pagination framework #28512

Closed

iRakson added 3 commits May 21, 2020 23:38

[SPARK-31642] Add Pagination Support for Structured Streaming Page

6d2b40e

fix

c5cc954

rebase and update

088c551

iRakson force-pushed the SPARK-31642 branch from 2486177 to 088c551 Compare May 21, 2020 19:11

sarutak approved these changes May 22, 2020

View reviewed changes

sarutak closed this in fbb3144 May 23, 2020

iRakson commented May 24, 2020

View reviewed changes

iRakson mentioned this pull request Jun 8, 2020

[SPARK-31983] Fix Sorting for duration column and make Status column sortable #28752

Closed

[SPARK-31642] Add Pagination Support for Structured Streaming Page #28485

[SPARK-31642] Add Pagination Support for Structured Streaming Page #28485

Uh oh!

Conversation

iRakson commented May 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

iRakson commented May 9, 2020

Uh oh!

HyukjinKwon commented May 10, 2020

Uh oh!

HyukjinKwon commented May 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented May 10, 2020

Uh oh!

iRakson commented May 10, 2020

Uh oh!

srowen commented May 10, 2020

Uh oh!

SparkQA commented May 10, 2020

Uh oh!

uncleGen commented May 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uncleGen commented May 11, 2020

Uh oh!

SparkQA commented May 11, 2020

Uh oh!

uncleGen left a comment

Choose a reason for hiding this comment

Uh oh!

uncleGen May 11, 2020

Choose a reason for hiding this comment

Uh oh!

iRakson May 11, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 11, 2020

Uh oh!

sarutak May 12, 2020

Choose a reason for hiding this comment

Uh oh!

uncleGen commented May 13, 2020

Uh oh!

iRakson commented May 21, 2020

Uh oh!

SparkQA commented May 22, 2020

Uh oh!

srowen commented May 22, 2020

Uh oh!

SparkQA commented May 22, 2020

Uh oh!

sarutak commented May 23, 2020

Uh oh!

iRakson commented May 23, 2020

Uh oh!

iRakson May 24, 2020

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented May 27, 2020

Uh oh!

iRakson commented May 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

iRakson commented May 9, 2020 •

edited

Loading

HyukjinKwon commented May 10, 2020 •

edited

Loading

uncleGen commented May 11, 2020 •

edited

Loading