Skip to content

[SPARK-15591] [WEBUI] Paginate Stage Table in Stages tab #13708

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nblintao wants to merge 6 commits into from
Closed

[SPARK-15591] [WEBUI] Paginate Stage Table in Stages tab #13708

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4255c09
paginate stage table in stages tab
nblintao Jun 16, 2016
4de6d68
add newline at end of AllStagesPage
nblintao Jun 16, 2016
246432d
fix problem of resitting other tables
nblintao Jun 21, 2016
7ccda86
fix the links to one-stage pages
nblintao Jun 22, 2016
4af0630
fix linking problems in history page & not display empty tables in po…
nblintao Jun 22, 2016
3a98c4d
fix nits to improve readability
nblintao Jul 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions core/src/main/scala/org/apache/spark/ui/PagedTable.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,7 @@ private[ui] trait PagedTable[T] {
Splitter
.on('&')
.trimResults()
.omitEmptyStrings()
.withKeyValueSeparator("=")
.split(querystring)
.asScala
Expand Down
21 changes: 10 additions & 11 deletions core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,22 +38,20 @@ private[ui] class AllStagesPage(parent: StagesTab) extends WebUIPage("") {
val numCompletedStages = listener.numCompletedStages
val failedStages = listener.failedStages.reverse.toSeq
val numFailedStages = listener.numFailedStages
val now = System.currentTimeMillis
val subPath = "stages"

val activeStagesTable =
new StageTableBase(activeStages.sortBy(_.submissionTime).reverse,
parent.basePath, parent.progressListener, isFairScheduler = parent.isFairScheduler,
killEnabled = parent.killEnabled)
new StageTableBase(request, activeStages, "activeStage", parent.basePath, subPath,
parent.progressListener, parent.isFairScheduler, parent.killEnabled, false)
val pendingStagesTable =
new StageTableBase(pendingStages.sortBy(_.submissionTime).reverse,
parent.basePath, parent.progressListener, isFairScheduler = parent.isFairScheduler,
killEnabled = false)
new StageTableBase(request, pendingStages, "pendingStage", parent.basePath, subPath,
parent.progressListener, parent.isFairScheduler, false, false)

@zsxwing zsxwing Jul 5, 2016 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could you add the parameter names for falses back to improve the readability?

val completedStagesTable =
new StageTableBase(completedStages.sortBy(_.submissionTime).reverse, parent.basePath,
parent.progressListener, isFairScheduler = parent.isFairScheduler, killEnabled = false)
new StageTableBase(request, completedStages, "completedStage", parent.basePath, subPath,
parent.progressListener, parent.isFairScheduler, false, false)

@zsxwing zsxwing Jul 5, 2016 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter names for falses

val failedStagesTable =
new FailedStageTable(failedStages.sortBy(_.submissionTime).reverse, parent.basePath,
parent.progressListener, isFairScheduler = parent.isFairScheduler)
new StageTableBase(request, failedStages, "failedStage", parent.basePath, subPath,
parent.progressListener, parent.isFairScheduler, false, true)

@zsxwing zsxwing Jul 5, 2016 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter names for false and true.


// For now, pool information is only accessible in live UIs
val pools = sc.map(_.getAllPools).getOrElse(Seq.empty[Schedulable])
Expand Down Expand Up @@ -136,3 +134,4 @@ private[ui] class AllStagesPage(parent: StagesTab) extends WebUIPage("") {
}
}
}

20 changes: 10 additions & 10 deletions core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,20 +229,20 @@ private[ui] class JobPage(parent: JobsTab) extends WebUIPage("job") {
}
}

val basePath = "jobs/job"

val activeStagesTable =
new StageTableBase(activeStages.sortBy(_.submissionTime).reverse,
parent.basePath, parent.jobProgresslistener, isFairScheduler = parent.isFairScheduler,
killEnabled = parent.killEnabled)
new StageTableBase(request, activeStages, "activeStage", parent.basePath,
basePath, parent.jobProgresslistener, parent.isFairScheduler, parent.killEnabled, false)

@zsxwing zsxwing Jul 5, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter name for false.

val pendingOrSkippedStagesTable =
new StageTableBase(pendingOrSkippedStages.sortBy(_.stageId).reverse,
parent.basePath, parent.jobProgresslistener, isFairScheduler = parent.isFairScheduler,
killEnabled = false)
new StageTableBase(request, pendingOrSkippedStages, "pendingStage", parent.basePath,
basePath, parent.jobProgresslistener, parent.isFairScheduler, false, false)

@zsxwing zsxwing Jul 5, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter names for falses.

val completedStagesTable =
new StageTableBase(completedStages.sortBy(_.submissionTime).reverse, parent.basePath,
parent.jobProgresslistener, isFairScheduler = parent.isFairScheduler, killEnabled = false)
new StageTableBase(request, completedStages, "completedStage", parent.basePath,
basePath, parent.jobProgresslistener, parent.isFairScheduler, false, false)

@zsxwing zsxwing Jul 5, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter names for falses.

val failedStagesTable =
new FailedStageTable(failedStages.sortBy(_.submissionTime).reverse, parent.basePath,
parent.jobProgresslistener, isFairScheduler = parent.isFairScheduler)
new StageTableBase(request, failedStages, "failedStage", parent.basePath,
basePath, parent.jobProgresslistener, parent.isFairScheduler, false, true)

@zsxwing zsxwing Jul 5, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter names for false and true.


val shouldShowActiveStages = activeStages.nonEmpty
val shouldShowPendingStages = !isComplete && pendingOrSkippedStages.nonEmpty
Expand Down
14 changes: 8 additions & 6 deletions core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,21 @@ private[ui] class PoolPage(parent: StagesTab) extends WebUIPage("pool") {
case Some(s) => s.values.toSeq
case None => Seq[StageInfo]()
}
val activeStagesTable = new StageTableBase(activeStages.sortBy(_.submissionTime).reverse,
parent.basePath, parent.progressListener, isFairScheduler = parent.isFairScheduler,
killEnabled = parent.killEnabled)
val shouldShowActiveStages = activeStages.nonEmpty
val activeStagesTable =
new StageTableBase(request, activeStages, "activeStage", parent.basePath, "stages/pool",
parent.progressListener, parent.isFairScheduler, parent.killEnabled, false)

@zsxwing zsxwing Jul 5, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the parameter name for false


// For now, pool information is only accessible in live UIs
val pools = sc.map(_.getPoolForName(poolName).getOrElse {
throw new IllegalArgumentException(s"Unknown poolname: $poolName")
}).toSeq
val poolTable = new PoolTable(pools, parent)

val content =
<h4>Summary </h4> ++ poolTable.toNodeSeq ++
<h4>{activeStages.size} Active Stages</h4> ++ activeStagesTable.toNodeSeq
var content = <h4>Summary </h4> ++ poolTable.toNodeSeq
if (shouldShowActiveStages) {
content ++= <h4>{activeStages.size} Active Stages</h4> ++ activeStagesTable.toNodeSeq
}

UIUtils.headerSparkPage("Fair Scheduler Pool: " + poolName, content, parent)
}
Expand Down
Loading