Skip to content

[SPARK-20966][WEB-UI][SQL]Table data is not sorted by startTime time desc, time is not formatted and redundant code in JDBC/ODBC Server page. #18186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
guoxiaolongzte wants to merge 37 commits into from
Closed

[SPARK-20966][WEB-UI][SQL]Table data is not sorted by startTime time desc, time is not formatted and redundant code in JDBC/ODBC Server page. #18186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
d383efb
[SPARK-20177]Document about compression way has some little detail ch…
Mar 31, 2017
3059013
[SPARK-20177] event log add a space
Apr 1, 2017
555cef8
'/applications/[app-id]/jobs' in rest api,status should be [running|s…
Apr 2, 2017
46bb1ad
Merge branch 'master' of https://github.com/apache/spark into SPARK-2…
Apr 5, 2017
0efb0dd
[SPARK-20218]'/applications/[app-id]/stages' in REST API,add descript…
Apr 5, 2017
0e37fde
[SPARK-20218] '/applications/[app-id]/stages/[stage-id]' in REST API,…
Apr 5, 2017
52641bb
Merge branch 'SPARK-20218'
Apr 7, 2017
d3977c9
Merge branch 'master' of https://github.com/apache/spark
Apr 8, 2017
137b90e
Merge branch 'master' of https://github.com/apache/spark
Apr 10, 2017
0fe5865
Merge branch 'SPARK-20190' of https://github.com/guoxiaolongzte/spark
Apr 10, 2017
cf6f42a
Merge branch 'master' of https://github.com/apache/spark
Apr 10, 2017
685cd6b
Merge branch 'master' of https://github.com/apache/spark
Apr 14, 2017
c716a92
Merge branch 'master' of https://github.com/apache/spark
Apr 17, 2017
679cec3
Merge branch 'master' of https://github.com/apache/spark
Apr 19, 2017
3c9387a
Merge branch 'master' of https://github.com/apache/spark
Apr 19, 2017
cb71f44
Merge branch 'master' of https://github.com/apache/spark
Apr 20, 2017
ce92a74
Merge branch 'master' of https://github.com/apache/spark
Apr 21, 2017
dd64342
Merge branch 'master' of https://github.com/apache/spark
Apr 21, 2017
bffd2bd
Merge branch 'master' of https://github.com/apache/spark
Apr 28, 2017
588d42a
Merge branch 'master' of https://github.com/apache/spark
Apr 28, 2017
4bbeee1
Merge branch 'master' of https://github.com/apache/spark
May 2, 2017
362e5ad
Merge branch 'master' of https://github.com/apache/spark
May 3, 2017
4ed5e00
增加代码
May 11, 2017
d871e01
Merge branch 'master' of https://github.com/apache/spark
May 11, 2017
a1f7759
Merge branch 'master' of https://github.com/apache/spark
May 11, 2017
dc56cdf
Merge branch 'master' of https://github.com/apache/spark
May 12, 2017
c999cf7
Merge branch 'master' of https://github.com/apache/spark
May 16, 2017
685dd8c
Merge branch 'master' of https://github.com/apache/spark
May 16, 2017
b744ceb
Merge branch 'master' of https://github.com/apache/spark
May 17, 2017
84a50f2
Merge branch 'master' of https://github.com/apache/spark
May 18, 2017
402b5f7
Merge branch 'master' of https://github.com/apache/spark
May 23, 2017
4867780
Merge branch 'master' of https://github.com/apache/spark
May 25, 2017
c1b1e38
Merge branch 'master' of https://github.com/apache/spark
May 26, 2017
85752db
Merge branch 'master' of https://github.com/apache/spark
May 31, 2017
172c872
Merge branch 'master' of https://github.com/apache/spark
Jun 1, 2017
6e60cc7
Merge branch 'master' of https://github.com/apache/spark
Jun 2, 2017
4c3dbeb
[SPARK-20966]Table data is not sorted by startTime time desc, time is…
Jun 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ...iftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ private[ui] class ThriftServerPage(parent: ThriftServerTab) extends WebUIPage(""
val table = if (numStatement > 0) {
val headerRow = Seq("User", "JobID", "GroupID", "Start Time", "Finish Time", "Duration",
"Statement", "State", "Detail")
val dataRows = listener.getExecutionList
val dataRows = listener.getExecutionList.sortBy(_.startTimestamp).reverse
Copy link
Copy Markdown
Member

@ajbozarth ajbozarth Jun 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What was this sorted by before? Was there a reason it was that way? If there wasn't, I'm ok with this change.

Copy link
Copy Markdown
Author

@guoxiaolongzte guoxiaolongzte Jun 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the ThriftServerSessionPage.scala, this sorted is order by startTimestamp desc.
But in the ThriftServerPage.scala, this sorted by before is default, this sorted is not order by startTimestamp desc.
So sort by time in reverse order is in line with spark UI style.


def generateDataRow(info: ExecutionInfo): Seq[Node] = {
val jobLink = info.jobId.map { id: String =>
Expand Down Expand Up @@ -142,7 +142,7 @@ private[ui] class ThriftServerPage(parent: ThriftServerTab) extends WebUIPage(""
val sessionList = listener.getSessionList
val numBatches = sessionList.size
val table = if (numBatches > 0) {
val dataRows = sessionList
val dataRows = sessionList.sortBy(_.startTimestamp).reverse
val headerRow = Seq("User", "IP", "Session ID", "Start Time", "Finish Time", "Duration",
"Total Execute")
def generateDataRow(session: SessionInfo): Seq[Node] = {
Expand Down
38 changes: 1 addition & 37 deletions ...er/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ private[ui] class ThriftServerSessionPage(parent: ThriftServerTab)
val timeSinceStart = System.currentTimeMillis() - startTime.getTime
<ul class ="unstyled">
<li>
<strong>Started at: </strong> {startTime.toString}
<strong>Started at: </strong> {formatDate(startTime)}
</li>
<li>
<strong>Time since start: </strong>{formatDurationVerbose(timeSinceStart)}
Expand Down Expand Up @@ -147,42 +147,6 @@ private[ui] class ThriftServerSessionPage(parent: ThriftServerTab)
<td>{errorSummary}{details}</td>
}

/** Generate stats of batch sessions of the thrift server program */
private def generateSessionStatsTable(): Seq[Node] = {
Copy link
Copy Markdown
Member

@ajbozarth ajbozarth Jun 2, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I can tell this has never been used, is this something that should be used rather than deleted? I have't worked in the ThiftSever UI much so I'm not sure myself.

Copy link
Copy Markdown
Author

@guoxiaolongzte guoxiaolongzte Jun 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the whole spark project, I did not find this method where it was used. When I remove this method to recompile the package test, I did not find any problems with this ThiftSever ui. SoI think can remove this method because it is redundant code. Thanks.

Copy link
Copy Markdown
Member

@srowen srowen Jun 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's also a method like this in ThriftServerPage.scala above. I guess we can remove it too. It's dead code.

Copy link
Copy Markdown
Author

@guoxiaolongzte guoxiaolongzte Jun 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, It's dead code.

Copy link
Copy Markdown
Author

@guoxiaolongzte guoxiaolongzte Jun 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@srowen
No, This method of generateSessionStatsTable in ThriftServerPage.scala is used. It is not dead code.

Copy link
Copy Markdown
Author

@guoxiaolongzte guoxiaolongzte Jun 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1

@srowen
This method implements the functions shown in the figure

val sessionList = listener.getSessionList
val numBatches = sessionList.size
val table = if (numBatches > 0) {
val dataRows =
sessionList.sortBy(_.startTimestamp).reverse.map ( session =>
Seq(
session.userName,
session.ip,
session.sessionId,
formatDate(session.startTimestamp),
formatDate(session.finishTimestamp),
formatDurationOption(Some(session.totalTime)),
session.totalExecution.toString
)
).toSeq
val headerRow = Seq("User", "IP", "Session ID", "Start Time", "Finish Time", "Duration",
"Total Execute")
Some(listingTable(headerRow, dataRows))
} else {
None
}

val content =
<h5>Session Statistics</h5> ++
<div>
<ul class="unstyled">
{table.getOrElse("No statistics have been generated yet.")}
</ul>
</div>

content
}


/**
* Returns a human-readable string representing a duration such as "5 second 35 ms"
*/
Expand Down