[SPARK-26714][CORE][WEBUI] Show 0 partition job in WebUI by deshanxiao · Pull Request #23637 · apache/spark

deshanxiao · 2019-01-24T09:34:37Z

What changes were proposed in this pull request?

When the job's partiton is zero, it will still get a jobid but not shown in ui. It's strange. This PR is to show this job in ui.

Example:
In bash:
mkdir -p /home/test/testdir

sc.textFile("/home/test/testdir")

Some logs:

19/01/24 17:26:19 INFO FileInputFormat: Total input paths to process : 0
19/01/24 17:26:19 INFO SparkContext: Starting job: collect at WordCount.scala:9
19/01/24 17:26:19 INFO DAGScheduler: Job 0 finished: collect at WordCount.scala:9, took 0.003735 s

How was this patch tested?

UT

deshanxiao · 2019-01-24T09:48:35Z

It looks like:

srowen · 2019-01-24T14:58:29Z

In this case is there actually an attempt 0?

Yes! The jobRow invokes the method to generate job page. Maybe I can use a if to handle the partition 0 job.

srowen

Hm, here's where I just don't know the code well enough to decide if that's the right place to return. I get that you're trying to account for the time taken to process the job, which waits on 0 tasks. Is that meaningful? conceptually it takes no time at all. It always succeeds too, right? can it even fail? it just seems a little weird to check this in two places, but might make sense, not sure.

deshanxiao · 2019-01-28T05:34:48Z

Yes, it takes no time at all and It always succeeds. Maybe using the same time in SparkListenerJobStart and SparkListenerJobEnd will be better. In addition, the method submitJob is invoked in two positions. I don't want to handle it twice. Hence, I think the best place to place the code is the method submitJob itself.

srowen

Yeah I was more comfortable with the original change, as you have it now.

deshanxiao · 2019-01-29T06:22:18Z

retest this please.

srowen · 2019-01-29T14:53:26Z


    val jobId = nextJobId.getAndIncrement()
    if (partitions.size == 0) {
+      val time = clock.getTimeMillis()


This is looking OK to me, though I wouldn't mind, say, @cloud-fan taking a quick look.

@srowen Thank you! @cloud-fan Could you give me some suggestions?

cloud-fan · 2019-01-29T15:10:07Z

LGTM, cc @gengliangwang

gengliangwang · 2019-01-29T16:24:38Z

After the changes, the UI is kind of confusing. Should we revise the wording "(Unknown Stage Name)"?

srowen · 2019-01-29T18:50:45Z

@gengliangwang actually now it will use the job.name as description if present, yes. That screenshot was from the original version of the PR.

SparkQA · 2019-01-29T19:45:57Z

Test build #4534 has finished for PR 23637 at commit 2185cb8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

deshanxiao · 2019-01-30T02:30:22Z

@gengliangwang @srowen Sorry, maybe I didn't make it clear. The job.name will be (Unknown Stage Name) in this case whatever changing PR or not. I change the original PR is to fit with the code logic.

We can see the default job.name will be sent in:

    // AppStatusListener.scala
    val lastStageInfo = event.stageInfos.sortBy(_.stageId).lastOption
    val lastStageName = lastStageInfo.map(_.name).getOrElse("(Unknown Stage Name)")
    val jobGroup = Option(event.properties)
      .flatMap { p => Option(p.getProperty(SparkContext.SPARK_JOB_GROUP_ID)) }
    val sqlExecutionId = Option(event.properties)
      .flatMap(p => Option(p.getProperty(SQL_EXECUTION_ID_KEY)).map(_.toLong))

    val job = new LiveJob(
      event.jobId,
      lastStageName,
      if (event.time > 0) Some(new Date(event.time)) else None,
      event.stageIds,
      jobGroup,
      numTasks,
      sqlExecutionId)
    liveJobs.put(event.jobId, job)
    liveUpdate(job, now)

private class LiveJob(
    val jobId: Int,
    name: String,
    val submissionTime: Option[Date],
    val stageIds: Seq[Int],
    jobGroup: Option[String],
    numTasks: Int,
    sqlExecutionId: Option[Long]) extends LiveEntity {

So, if necessary， we can set job.name in two methods:

1.Add more info in SparkListenerJobStart
2.Constructing a empty stage contains a callsite in StageInfo?

But it will be more complex and get some compatibility problem.

In the end, I change it to original. Mybe the UI is kind of confusing. This is acceptable because the job description in here places the last stage info originally. In this case, we have no stage. So showing "Unknown Stage Name" is acceptable I think.

deshanxiao · 2019-01-30T02:32:07Z

Here is the lastest PR screenshot:

srowen · 2019-01-30T02:52:42Z

Oh I see. Well it's consistent with the current logic which uses job.name in all cases as a fallback. I think this is OK; I'm not sure what other placeholder string would be more meaningful.

deshanxiao · 2019-01-30T02:59:44Z

@srowen Yes, I argee with you. @gengliangwang Could you give me some suggestions? What placeholder string more meaningful do you think?

gengliangwang · 2019-01-30T05:16:28Z

Hi @deshanxiao ,
yes, the description (Unknown Stage Name) is for all the jobs that have 0 stages, which is not directly related to this PR.
My point is, can we also update this Job description? In the job page, it is confusing to me that the description is (Unknown Stage Name). I would suggest to revise it as Unknown Job without stages or Unknown Job.
But this is just personal opinion. We can go without it. This PR LGTM overall.

deshanxiao · 2019-01-30T06:22:04Z

I get it. Thank you @gengliangwang! I think Unknown Job without stages will be better. I will change it. Thanks!

deshanxiao · 2019-01-30T08:34:46Z

Retest please.

gengliangwang

LGTM

cloud-fan · 2019-01-31T03:15:34Z

ok to test

SparkQA · 2019-01-31T08:05:02Z

Test build #101929 has finished for PR 23637 at commit 0c1ea7e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-01-31T08:21:38Z

retest this please

SparkQA · 2019-01-31T13:07:18Z

Test build #101943 has finished for PR 23637 at commit 0c1ea7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2019-02-02T00:38:40Z

Merged to master

## What changes were proposed in this pull request? When the job's partiton is zero, it will still get a jobid but not shown in ui. It's strange. This PR is to show this job in ui. Example: In bash: mkdir -p /home/test/testdir sc.textFile("/home/test/testdir") Some logs: ``` 19/01/24 17:26:19 INFO FileInputFormat: Total input paths to process : 0 19/01/24 17:26:19 INFO SparkContext: Starting job: collect at WordCount.scala:9 19/01/24 17:26:19 INFO DAGScheduler: Job 0 finished: collect at WordCount.scala:9, took 0.003735 s ``` ## How was this patch tested? UT Closes apache#23637 from deshanxiao/spark-26714. Authored-by: xiaodeshan <xiaodeshan@xiaomi.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>

srowen reviewed Jan 24, 2019

View reviewed changes

update

861db42

srowen requested changes Jan 25, 2019

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala

Comment thread core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala Outdated

xiaodeshan added 2 commits January 28, 2019 00:12

update

81f0340

update

5140145

srowen requested changes Jan 27, 2019

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala

update

1b04a2b

srowen reviewed Jan 28, 2019

View reviewed changes

update

e6ddecf

srowen requested changes Jan 28, 2019

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala Outdated

update

2185cb8

srowen reviewed Jan 29, 2019

View reviewed changes

update job default name

df719c5

gengliangwang reviewed Jan 30, 2019

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/status/AppStatusListener.scala Outdated

update

97112e5

gengliangwang approved these changes Jan 30, 2019

View reviewed changes

srowen reviewed Jan 30, 2019

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/status/AppStatusListener.scala Outdated

update

0c1ea7e

srowen approved these changes Feb 2, 2019

View reviewed changes

srowen closed this in a0faabf Feb 2, 2019

squito mentioned this pull request Mar 15, 2019

[SPARK-27164][Core] RDD.countApprox on empty RDDs schedules jobs which never complete #24100

Closed

Conversation

deshanxiao commented Jan 24, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

deshanxiao commented Jan 24, 2019

Uh oh!

Uh oh!

srowen Jan 24, 2019

Choose a reason for hiding this comment

Uh oh!

deshanxiao Jan 25, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

deshanxiao commented Jan 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deshanxiao commented Jan 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen Jan 29, 2019

Choose a reason for hiding this comment

Uh oh!

deshanxiao Jan 29, 2019

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jan 29, 2019

Uh oh!

gengliangwang commented Jan 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Jan 29, 2019

Uh oh!

SparkQA commented Jan 29, 2019

Uh oh!

deshanxiao commented Jan 30, 2019

Uh oh!

deshanxiao commented Jan 30, 2019

Uh oh!

srowen commented Jan 30, 2019

Uh oh!

deshanxiao commented Jan 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Jan 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deshanxiao commented Jan 30, 2019

Uh oh!

deshanxiao commented Jan 30, 2019

Uh oh!

Uh oh!

gengliangwang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cloud-fan commented Jan 31, 2019

Uh oh!

SparkQA commented Jan 31, 2019

Uh oh!

cloud-fan commented Jan 31, 2019

Uh oh!

SparkQA commented Jan 31, 2019

Uh oh!

srowen commented Feb 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

deshanxiao commented Jan 28, 2019 •

edited

Loading

deshanxiao commented Jan 29, 2019 •

edited

Loading

gengliangwang commented Jan 29, 2019 •

edited

Loading

deshanxiao commented Jan 30, 2019 •

edited

Loading

gengliangwang commented Jan 30, 2019 •

edited

Loading