Skip to content

[SPARK-29559][Webui] Support pagination for JDBC/ODBC Server page #26215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
shahidki31 wants to merge 1 commit into from
Closed

[SPARK-29559][Webui] Support pagination for JDBC/ODBC Server page #26215

shahidki31 wants to merge 1 commit into from

Conversation

@shahidki31
Copy link
Contributor

@shahidki31 shahidki31 commented Oct 22, 2019 •
edited
Loading

What changes were proposed in this pull request?

Supports pagination for SQL Statisitcs table in the JDBC/ODBC tab using existing Spark pagination framework.

Why are the changes needed?

It will easier for user to analyse the table and it may fix the potential issues like oom while loading the page, that may occur similar to the SQL page (refer #22645)

Does this PR introduce any user-facing change?

There will be no change in the SQLStatistics table in JDBC/ODBC server page execpt pagination support.

How was this patch tested?

Manually verified.

Before PR:
Screenshot 2019-10-22 at 11 37 29 PM

After PR:

Screenshot 2019-10-22 at 10 33 00 PM

@shahidki31 shahidki31 changed the title [SPARK-29559][Webui]Support pagination for JDBC/ODBC Server tab [SPARK-29559][Webui]Support pagination for JDBC/ODBC Server page Oct 22, 2019
@SparkQA
Copy link

SparkQA commented Oct 22, 2019

Test build #112481 has finished for PR 26215 at commit 8747a9c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@shahidki31
Copy link
Contributor Author

shahidki31 commented Oct 22, 2019

cc @wangyum @srowen Kindly review

@srowen
Copy link
Member

srowen commented Oct 23, 2019

For my reference, is most of the change here just refactoring the code? it's hard to tell what changed. Maybe you can highlight a few key parts that have changed.
Also does this generally follow how other tables are paginated or is it different?

@shahidki31
Copy link
Contributor Author

shahidki31 commented Oct 23, 2019

There is an existing framework in Spark for pagination of WebUI tables. (Refer PagedTable.scala).
Currently, Jobs Page, All Stages Page , SQL page are following the framework by extending the classes. I have done the similar to these changes here.

Basically we need to create 2 classes that extends to PagedDataSource and PagedTable .

In this PR: SqlStatsPagedTable -> PagedTable and SqlStatsTableDataSource -> PagedDataSource

Also class SqlStatsTableRow helper class for sorting purpose and just hold the data.

Other changes are just refactoring. I think, I followed similar to the PR: #22645

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-29559][Webui]Support pagination for JDBC/ODBC Server page [SPARK-29559][Webui] Support pagination for JDBC/ODBC Server page Oct 23, 2019
@srowen srowen closed this in 76d4beb Oct 24, 2019
@srowen
Copy link
Member

srowen commented Oct 24, 2019

Merged to master

@shahidki31
Copy link
Contributor Author

shahidki31 commented Oct 24, 2019

Thanks @srowen . I will raise followup PRs for other improvements in JDBC/ODBC server page. https://jira.apache.org/jira/browse/SPARK-29588

@shahidki31 shahidki31 mentioned this pull request Oct 24, 2019
Closed
@AngersZhuuuu AngersZhuuuu mentioned this pull request Oct 25, 2019
Closed
srowen pushed a commit that referenced this pull request Oct 26, 2019
@shahidki31 @srowen
[SPARK-29589][WEBUI] Support pagination for sqlstats session table in…
077fb99 
… JDBC/ODBC Session page

### What changes were proposed in this pull request?
In the PR #26215, we supported pagination for sqlstats table in JDBC/ODBC server page. In this PR, we are extending the support of pagination to sqlstats session table by making use of existing pagination classes in #26215.

### Why are the changes needed?
Support pagination for sqlsessionstats table in JDBC/ODBC server page in the WEBUI. It will easier for user to analyse the table and it may fix the potential issues like oom while loading the page, that may occur similar to the SQL page (refer #22645)

### Does this PR introduce any user-facing change?
There will be no change in the sqlsessionstats table in JDBC/ODBC server page execpt pagination support.

### How was this patch tested?
Manually verified.

Before:

![Screenshot 2019-10-24 at 11 32 27 PM](https://user-images.githubusercontent.com/23054875/67512507-96715000-f6b6-11e9-9f1f-ab1877eb24e6.png)

After:

![Screenshot 2019-10-24 at 10 58 53 PM](https://user-images.githubusercontent.com/23054875/67512314-295dba80-f6b6-11e9-9e3e-dd50c6e62fe9.png)

Closes #26246 from shahidki31/SPARK_29589.

Authored-by: shahid <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
dengziming pushed a commit to dengziming/spark that referenced this pull request Oct 28, 2019
[SPARK-29589][WEBUI] Support pagination for sqlstats session table in…
a6ddbbf 
… JDBC/ODBC Session page

In the PR apache#26215, we supported pagination for sqlstats table in JDBC/ODBC server page. In this PR, we are extending the support of pagination to sqlstats session table by making use of existing pagination classes in apache#26215.

Support pagination for sqlsessionstats table in JDBC/ODBC server page in the WEBUI. It will easier for user to analyse the table and it may fix the potential issues like oom while loading the page, that may occur similar to the SQL page (refer apache#22645)

There will be no change in the sqlsessionstats table in JDBC/ODBC server page execpt pagination support.

Manually verified.

Before:

![Screenshot 2019-10-24 at 11 32 27 PM](https://user-images.githubusercontent.com/23054875/67512507-96715000-f6b6-11e9-9f1f-ab1877eb24e6.png)

After:

![Screenshot 2019-10-24 at 10 58 53 PM](https://user-images.githubusercontent.com/23054875/67512314-295dba80-f6b6-11e9-9e3e-dd50c6e62fe9.png)

Closes apache#26246 from shahidki31/SPARK_29589.

Authored-by: shahid <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
Signed-off-by: dengziming <[email protected]>
juliuszsompolski
juliuszsompolski reviewed Nov 14, 2019
View reviewed changes
Copy link
Contributor

@juliuszsompolski juliuszsompolski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @srowen @shahidki31 - did this PR remove the table sorting capabilities?

...thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala
<td>{formatDate(info.startTimestamp)}</td>
<td>{if (info.finishTimestamp > 0) formatDate(info.finishTimestamp)}</td>
<td>{if (info.closeTimestamp > 0) formatDate(info.closeTimestamp)}</td>
<td sorttable_customkey={info.totalTime(info.finishTimestamp).toString}>
Copy link
Contributor

@juliuszsompolski juliuszsompolski Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shahidki31 @srowen Is the new table not sortable now?

Copy link
Contributor Author

@shahidki31 shahidki31 Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is sortable, as we are sorting on the raw data in the Ordering method.

...thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala
{if (info.finishTimestamp > 0) formatDate(info.finishTimestamp)}
</td>
<td>
{if (info.closeTimestamp > 0) formatDate(info.closeTimestamp)}
Copy link
Contributor

@juliuszsompolski juliuszsompolski Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should sorttable_customkey be added to these, like in the code that was removed above?

Copy link
Contributor Author

@shahidki31 shahidki31 Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as #26215 (comment) Thanks

Copy link
Contributor

@juliuszsompolski juliuszsompolski Nov 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. Sorry for my confusion.

@shahidki31 shahidki31 deleted the jdbcPagination branch November 17, 2019 11:14
@iRakson iRakson mentioned this pull request Dec 4, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowen srowen srowen approved these changes

+1 more reviewer

@juliuszsompolski juliuszsompolski juliuszsompolski left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

WEB UI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@shahidki31 @SparkQA @srowen @juliuszsompolski @dongjoon-hyun