Skip to content

[SPARK-24851][UI] Map a Stage ID to it's Associated Job ID #21809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pgandhi999 wants to merge 9 commits into from
Closed

[SPARK-24851][UI] Map a Stage ID to it's Associated Job ID #21809

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7be0520
[SPARK-24851] : Map a Stage ID to it's Associated Job ID in UI
Jul 18, 2018
3151a62
[SPARK-24851] : Using both stage id as well as the attempt id to fetc…
Jul 23, 2018
a50e8b1
[SPARK-24851] : Avoiding extra store lookup and adding href links
Jul 25, 2018
d57e6dc
[SPARK-24851] : Fixing Scalastyle Tests
Jul 25, 2018
3a06b87
[SPARK-24851] : Adding exception handling part to avoid NoSuchElement…
Jul 26, 2018
52b08b2
[SPARK-24851] : Addressing reviews July 27
Jul 27, 2018
9eff537
[SPARK-24851] : Addressing Reviews August 9
Aug 9, 2018
3b73840
[SPARK-24851] : Addressing reviews September 28, 2018
Sep 28, 2018
0be099a
[SPARK-24851] : Addressing Reviews October 5, 2018
Oct 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ private[spark] class AppStatusStore(
}.toSeq
}

def getJobIdsAssociatedWithStage(stageId: Int, stageAttemptId: Int): Seq[Int] = {
val stageKey = Array(stageId, stageAttemptId)
val jobIds = store.read(classOf[StageDataWrapper], stageKey).jobIds.toSeq
jobIds
}

def lastStageAttempt(stageId: Int): v1.StageData = {
val it = store.view(classOf[StageDataWrapper])
.index("stageId")
Expand Down
8 changes: 8 additions & 0 deletions core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
return UIUtils.headerSparkPage(request, stageHeader, content, parent)
}

val stageJobIds = parent.store.getJobIdsAssociatedWithStage(stageId, stageAttemptId)
Copy link
Member

@gengliangwang gengliangwang Jul 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala#L109 there is a query for the stage data already. We can reduce the query to the store here.

Copy link
Member

@gengliangwang gengliangwang Jul 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g. we can add a function to return the whole StageDataWrapper in AppStatusStore

Copy link
Author

@pgandhi999 pgandhi999 Jul 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, I have fixed it. Request you to have a look.


val localitySummary = store.localitySummary(stageData.stageId, stageData.attemptId)

val totalTasks = taskCount(stageData)
Expand Down Expand Up @@ -182,6 +184,12 @@ private[ui] class StagePage(parent: StagesTab, store: AppStatusStore) extends We
{Utils.bytesToString(stageData.diskBytesSpilled)}
</li>
}}
{if (!stageJobIds.isEmpty) {
Copy link
Contributor

@abellina abellina Sep 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could throw an NPE if stageDataTuple is None

scala> var x:Seq[Int] = null
x: Seq[Int] = null

scala> x.isEmpty
java.lang.NullPointerException

Copy link
Author

@pgandhi999 pgandhi999 Sep 28, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since, we are returning from the code if it is None, it should never reach there as far as I can tell.

<li>
<strong>Associated Job Ids: </strong>
{stageJobIds}
Copy link
Member

@gengliangwang gengliangwang Jul 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make it href link?

Copy link
Author

@pgandhi999 pgandhi999 Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem here is that the stage could also have multiple job ids, in that case, we get a bunch of them. Do you want a generic link instead that will take you to the jobs page? Let me know what you think.

Copy link
Member

@gengliangwang gengliangwang Jul 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion is to map each job id as a href link.

Copy link
Author

@pgandhi999 pgandhi999 Jul 25, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did that and updated the screenshot. Thank you.

</li>
}}
</ul>
</div>

Expand Down