[SPARK-32125][UI] Support get taskList by status in Web UI and SHS Rest API #28942

warrenzhu25 · 2020-06-28T23:22:09Z

What changes were proposed in this pull request?

Support fetching taskList by status as below:

/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/taskList?status=failed

Why are the changes needed?

When there're large number of tasks in one stage, current api is hard to get taskList by status

Does this PR introduce any user-facing change?

Yes. Updated monitoring doc.

How was this patch tested?

Added tests in HistoryServerSuite

…st API

sarutak · 2020-06-29T05:43:08Z

Hi @warrenzhu25 , thank you for your contribution.
This PR seems to add a new feature so could you add a testcase for it?
You can find tests for the status API in UISeleniumSuite and HistoryServerSuite.

warrenzhu25 · 2020-06-29T22:20:59Z

Hi @warrenzhu25 , thank you for your contribution.
This PR seems to add a new feature so could you add a testcase for it?
You can find tests for the status API in UISeleniumSuite and HistoryServerSuite.

Added UT, but it seems doc build failed. It seems unrelated with my change.

sarutak · 2020-06-30T02:35:51Z

ok to test.

sarutak · 2020-06-30T02:36:12Z

cc: @squito

SparkQA · 2020-06-30T22:26:03Z

Test build #124655 has finished for PR 28942 at commit c10e38f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-01T01:13:54Z

retest this please.

SparkQA · 2020-07-01T04:39:09Z

Test build #124714 has finished for PR 28942 at commit c10e38f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-07-01T13:15:11Z

core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala

-      @DefaultValue("ID") @QueryParam("sortBy") sortBy: TaskSorting): Seq[TaskData] = {
-    withUI(_.store.taskList(stageId, stageAttemptId, offset, length, sortBy))
+      @DefaultValue("ID") @QueryParam("sortBy") sortBy: TaskSorting,
+      @QueryParam("status") statuses: JList[TaskStatus]): Seq[TaskData] = {


Why do we need to make it as JList here? How about just TaskStatus?

Ok, I'm also considering using TaskStatus. The reason I use JList is to make it consistent with stageList as below:

@GET def stageList(@QueryParam("status") statuses: JList[StageStatus]): Seq[StageData] = { withUI(_.store.stageList(statuses)) }

Oh, sorry I was not aware of this. Please change it back to JList to make it consistent.

…parameter

SparkQA · 2020-07-01T20:59:22Z

Test build #124801 has finished for PR 28942 at commit 2f55b82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…request parameter" This reverts commit 2f55b82.

SparkQA · 2020-07-02T03:54:36Z

Test build #124833 has finished for PR 28942 at commit e8e5d6a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-07-02T05:52:08Z

retest this please.

sarutak · 2020-07-02T06:34:23Z

Jenkins will restart around 0:00 PDT so we'll need to let Jenkins run again.

SparkQA · 2020-07-02T07:05:02Z

Test build #124868 has finished for PR 28942 at commit e8e5d6a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-02T07:11:40Z

retest this please.

SparkQA · 2020-07-02T07:20:30Z

Test build #124876 has finished for PR 28942 at commit e8e5d6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-02T07:24:19Z

retest this please.

SparkQA · 2020-07-02T07:27:49Z

Test build #124879 has finished for PR 28942 at commit e8e5d6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-02T07:39:06Z

retest this please.

SparkQA · 2020-07-02T07:42:46Z

Test build #124881 has finished for PR 28942 at commit e8e5d6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-02T07:49:24Z

All the dependency failure happens on jenkins-worker4.
It seems worker4 has .m2 corruption issue.

sarutak · 2020-07-02T08:15:42Z

I've reported the .m2 corruption issue (https://issues.apache.org/jira/browse/SPARK-32153#).
So let's try again.

sarutak · 2020-07-02T08:29:23Z

retest this please.

SparkQA · 2020-07-02T08:35:41Z

Test build #124885 has finished for PR 28942 at commit e8e5d6a.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-06T17:58:12Z

retest this please.

SparkQA · 2020-07-06T18:34:43Z

Test build #125114 has finished for PR 28942 at commit e8e5d6a.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2020-07-06T18:52:22Z

retest this please.

SparkQA · 2020-07-06T23:13:15Z

Test build #125117 has finished for PR 28942 at commit e8e5d6a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

warrenzhu25 · 2020-07-10T15:13:23Z

retest this please.

sarutak · 2020-07-13T15:23:16Z

retest this please.

SparkQA · 2020-07-13T18:10:02Z

Test build #125777 has finished for PR 28942 at commit e8e5d6a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

warrenzhu25 · 2020-07-15T04:49:20Z

@gengliangwang Tests passed, could you help merge this?

gengliangwang · 2020-07-16T03:29:46Z

Thanks, merging to master

probot-autolabeler bot added CORE DOCS labels Jun 28, 2020

[SPARK-32125][UI] Support get taskList by status in Web UI and SHS Re…

9cc5e3f

…st API

warrenzhu25 force-pushed the SPARK-32125 branch from ff001db to 9cc5e3f Compare June 28, 2020 23:25

[SPARK-32125][UI] Add UT in HistoryServerSuite

c10e38f

gengliangwang reviewed Jul 1, 2020

View reviewed changes

[SPARK-32125] Use TaskStatus instead of JList[TaskStatus] as request …

2f55b82

…parameter

Revert "[SPARK-32125] Use TaskStatus instead of JList[TaskStatus] as …

e8e5d6a

…request parameter" This reverts commit 2f55b82.

gengliangwang closed this in db47c6e Jul 16, 2020

warrenzhu25 deleted the SPARK-32125 branch June 11, 2022 17:12

[SPARK-32125][UI] Support get taskList by status in Web UI and SHS Rest API #28942

[SPARK-32125][UI] Support get taskList by status in Web UI and SHS Rest API #28942

Uh oh!

Conversation

warrenzhu25 commented Jun 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak commented Jun 29, 2020

Uh oh!

warrenzhu25 commented Jun 29, 2020

Uh oh!

sarutak commented Jun 30, 2020

Uh oh!

sarutak commented Jun 30, 2020

Uh oh!

SparkQA commented Jun 30, 2020

Uh oh!

sarutak commented Jul 1, 2020

Uh oh!

SparkQA commented Jul 1, 2020

Uh oh!

gengliangwang Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

warrenzhu25 Jul 1, 2020

Choose a reason for hiding this comment

Uh oh!

gengliangwang Jul 2, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 1, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

gengliangwang commented Jul 2, 2020

Uh oh!

sarutak commented Jul 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

sarutak commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

sarutak commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

sarutak commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

sarutak commented Jul 2, 2020

Uh oh!

sarutak commented Jul 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented Jul 2, 2020

Uh oh!

SparkQA commented Jul 2, 2020

Uh oh!

sarutak commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

sarutak commented Jul 6, 2020

Uh oh!

SparkQA commented Jul 6, 2020

Uh oh!

warrenzhu25 commented Jul 10, 2020

Uh oh!

sarutak commented Jul 13, 2020

Uh oh!

SparkQA commented Jul 13, 2020

warrenzhu25 commented Jun 28, 2020 •

edited

Loading

gengliangwang Jul 1, 2020 •

edited

Loading

sarutak commented Jul 2, 2020 •

edited

Loading

sarutak commented Jul 2, 2020 •

edited

Loading