[SPARK-23431][CORE] Expose stage level peak executor metrics via REST API #29020

imback82 · 2020-07-07T07:22:14Z

What changes were proposed in this pull request?

Note that this PR is forked from #23340 originally written by @edwinalu.

This PR proposes to expose the peak executor metrics at the stage level via the REST APIs:

/applications/<application_id>/stages/: peak values of executor metrics for each stage
/applications/<application_id>/stages/<stage_id>/< stage_attempt_id >: peak values of executor metrics for each executor for the stage, followed by peak values of executor metrics for the stage

Why are the changes needed?

The stage level peak executor metrics can help better understand your application's resource utilization.

Does this PR introduce any user-facing change?

For the /applications/<application_id>/stages/ API, you will see the following new info for each stage:

  "peakExecutorMetrics" : {
    "JVMHeapMemory" : 213367864,
    "JVMOffHeapMemory" : 189011656,
    "OnHeapExecutionMemory" : 0,
    "OffHeapExecutionMemory" : 0,
    "OnHeapStorageMemory" : 2133349,
    "OffHeapStorageMemory" : 0,
    "OnHeapUnifiedMemory" : 2133349,
    "OffHeapUnifiedMemory" : 0,
    "DirectPoolMemory" : 282024,
    "MappedPoolMemory" : 0,
    "ProcessTreeJVMVMemory" : 0,
    "ProcessTreeJVMRSSMemory" : 0,
    "ProcessTreePythonVMemory" : 0,
    "ProcessTreePythonRSSMemory" : 0,
    "ProcessTreeOtherVMemory" : 0,
    "ProcessTreeOtherRSSMemory" : 0,
    "MinorGCCount" : 13,
    "MinorGCTime" : 115,
    "MajorGCCount" : 4,
    "MajorGCTime" : 339
  }

For the /applications/<application_id>/stages/<stage_id>/<stage_attempt_id> API, you will see the following new info for each executor under executorSummary:

  "peakMemoryMetrics" : {
    "JVMHeapMemory" : 0,
    "JVMOffHeapMemory" : 0,
    "OnHeapExecutionMemory" : 0,
    "OffHeapExecutionMemory" : 0,
    "OnHeapStorageMemory" : 0,
    "OffHeapStorageMemory" : 0,
    "OnHeapUnifiedMemory" : 0,
    "OffHeapUnifiedMemory" : 0,
    "DirectPoolMemory" : 0,
    "MappedPoolMemory" : 0,
    "ProcessTreeJVMVMemory" : 0,
    "ProcessTreeJVMRSSMemory" : 0,
    "ProcessTreePythonVMemory" : 0,
    "ProcessTreePythonRSSMemory" : 0,
    "ProcessTreeOtherVMemory" : 0,
    "ProcessTreeOtherRSSMemory" : 0,
    "MinorGCCount" : 0,
    "MinorGCTime" : 0,
    "MajorGCCount" : 0,
    "MajorGCTime" : 0
  }

, and the following at the stage level:

"peakExecutorMetrics" : {
    "JVMHeapMemory" : 213367864,
    "JVMOffHeapMemory" : 189011656,
    "OnHeapExecutionMemory" : 0,
    "OffHeapExecutionMemory" : 0,
    "OnHeapStorageMemory" : 2133349,
    "OffHeapStorageMemory" : 0,
    "OnHeapUnifiedMemory" : 2133349,
    "OffHeapUnifiedMemory" : 0,
    "DirectPoolMemory" : 282024,
    "MappedPoolMemory" : 0,
    "ProcessTreeJVMVMemory" : 0,
    "ProcessTreeJVMRSSMemory" : 0,
    "ProcessTreePythonVMemory" : 0,
    "ProcessTreePythonRSSMemory" : 0,
    "ProcessTreeOtherVMemory" : 0,
    "ProcessTreeOtherRSSMemory" : 0,
    "MinorGCCount" : 13,
    "MinorGCTime" : 115,
    "MajorGCCount" : 4,
    "MajorGCTime" : 339
  }

How was this patch tested?

Added tests.

SparkQA · 2020-07-07T07:33:04Z

Test build #125188 has finished for PR 29020 at commit bbeab0e.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

itamarst · 2020-07-07T14:51:48Z

@imback82 Thank you!

imback82 · 2020-07-07T18:30:22Z

retest this please

SparkQA · 2020-07-07T18:33:16Z

Test build #125244 has finished for PR 29020 at commit bbeab0e.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-07T23:07:24Z

Test build #125246 has finished for PR 29020 at commit cc4018a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-08T06:28:09Z

Test build #125278 has finished for PR 29020 at commit 2d5b73d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-08T06:34:07Z

retest this please

SparkQA · 2020-07-08T07:05:02Z

Test build #125309 has finished for PR 29020 at commit 2d5b73d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-08T07:38:37Z

retest this please

SparkQA · 2020-07-08T07:43:06Z

Test build #125328 has started for PR 29020 at commit 2d5b73d.

imback82 · 2020-07-08T17:03:40Z

Retest this please

SparkQA · 2020-07-08T18:02:54Z

Test build #125387 has finished for PR 29020 at commit 2d5b73d.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-08T18:05:03Z

retest this please

SparkQA · 2020-07-08T18:35:48Z

Test build #125390 has finished for PR 29020 at commit 2d5b73d.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-09T18:16:25Z

retest this please

SparkQA · 2020-07-09T21:57:40Z

Test build #125508 has finished for PR 29020 at commit 2d5b73d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-09T22:01:17Z

retest this please

SparkQA · 2020-07-10T01:18:09Z

Test build #125517 has finished for PR 29020 at commit 2d5b73d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-10T02:50:21Z

retest this please

SparkQA · 2020-07-10T07:05:03Z

Test build #125544 has finished for PR 29020 at commit 2d5b73d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-10T19:29:28Z

retest this please

SparkQA · 2020-07-10T21:40:19Z

Test build #125636 has finished for PR 29020 at commit 2d5b73d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-17T19:43:51Z

retest this please

SparkQA · 2020-07-17T23:48:06Z

Test build #126070 has finished for PR 29020 at commit 895f5fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-07-21T18:48:10Z

@gengliangwang gentle ping

gengliangwang · 2020-07-24T18:09:49Z

cc @srowen @vanzin

dongjoon-hyun · 2020-07-24T18:26:09Z

Retest this please.

dongjoon-hyun

Hi, @imback82 .
For the following your comment, please add an empty commit with author @edwinalu . Then, Apache Spark script will make him as a co-author.

 I will also ask a reviewing committer to put @edwinalu as a co-author if that's the practice when a PR is "inherited". Thanks!

To committers, please don't merge this PR until this has a correct authorship.

dongjoon-hyun · 2020-07-24T18:57:30Z

To @gengliangwang, please try to recommend the authorship commit explicitly in the Apache Spark community. Otherwise, it's forgot frequently during merging steps. Thanks!

imback82 · 2020-07-24T19:21:47Z

Thanks @dongjoon-hyun for the reminder! I added @edwinalu as a co-author with:

git commit --allow-empty --author="edwinalu <[email protected]>" -m "Add co-author"

edwinalu · 2020-07-24T19:22:21Z

@imback82 , thanks for all your work on this!

SparkQA · 2020-07-24T21:07:59Z

Test build #126509 has finished for PR 29020 at commit 895f5fd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-24T22:09:36Z

Test build #126512 has finished for PR 29020 at commit ddcb772.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wypoon · 2020-07-25T02:08:32Z

In the PR description,

"This PR proposes to expose the peak executor metrics at the stage level via the REST API (/applications/<application_id>/stages/ or /applications/<application_id>/stages/<stage_id>/<stage_attemp_id>)."

there is a typo; "<stage_attemp_id>" should be "<stage_attempt_id>".

It might be helpful to clarify that the peak values of executor metrics for the stage are shown for each stage in the first (/applications/<application_id>/stages/) and peak values of executor metrics for each executor for the stage are shown for the second /applications/<application_id>/stages/<stage_id>/<stage_attempt_id>). In other words, the first shows per-stage peaks, and the second shows per-executor per-stage peaks (for just that stage).
At least, this is my understanding of what is intended.
Edwina's PR description has

"Peak values for metrics are show for each stage. For executor summaries for each stage, the peak values per executor are also shown."

imback82 · 2020-07-25T02:54:52Z

Thanks @wypoon for the suggestion. I updated the description, and hopefully that clarifies things a bit more.

gengliangwang · 2020-07-25T06:01:13Z

@dongjoon-hyun Thanks for the reminder!

gengliangwang · 2020-08-04T03:43:04Z

retest this please.

gengliangwang · 2020-08-04T03:57:26Z

@imback82 I saw this error from github action tests:

ERROR: this R is version 3.4.4, package 'SparkR' requires R >= 3.5
[error] running /home/runner/work/spark/spark/R/install-dev.sh

Could you rebase the PR to the latest master? I will merge it once the tests are passed. Thanks!

SparkQA · 2020-08-04T07:02:48Z

Test build #127024 has finished for PR 29020 at commit ddcb772.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-04T07:05:03Z

Test build #127027 has finished for PR 29020 at commit 2115007.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

imback82 · 2020-08-04T07:10:33Z

retest this please

SparkQA · 2020-08-04T09:42:08Z

Test build #127043 has finished for PR 29020 at commit 2115007.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-08-04T13:10:18Z

Thanks, merging to master

dongjoon-hyun · 2020-08-04T14:02:42Z

Thank you, @imback82 and @gengliangwang !

imback82 · 2020-08-04T16:47:13Z

Thanks everyone!

imback82 added 2 commits July 6, 2020 22:17

initial commit

c8ff332

update tests

bbeab0e

imback82 mentioned this pull request Jul 7, 2020

[SPARK-23431][CORE] Expose the new executor memory metrics at the stage level #23340

Closed

update .rat-excludes

cc4018a

update test output

2d5b73d

probot-autolabeler bot added BUILD CORE WEB UI labels Jul 8, 2020

gengliangwang approved these changes Jul 24, 2020

View reviewed changes

dongjoon-hyun requested changes Jul 24, 2020

View reviewed changes

imback82 and others added 2 commits July 24, 2020 12:18

Merge branch 'master' into metrics

e236785

Add co-author

ddcb772

itamarst mentioned this pull request Jul 28, 2020

Tracking peak memory usage linkedin/dr-elephant#689

Closed

Merge branch 'master' into metrics

2115007

gengliangwang closed this in 171b7d5 Aug 4, 2020

gengliangwang mentioned this pull request Dec 4, 2020

[SPARK-26341][WEBUI]Expose executor memory metrics at the stage level, in the Stages tab #30573

Closed

[SPARK-23431][CORE] Expose stage level peak executor metrics via REST API #29020

[SPARK-23431][CORE] Expose stage level peak executor metrics via REST API #29020

Uh oh!

Conversation

imback82 commented Jul 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jul 7, 2020

Uh oh!

itamarst commented Jul 7, 2020

Uh oh!

imback82 commented Jul 7, 2020

Uh oh!

SparkQA commented Jul 7, 2020

Uh oh!

SparkQA commented Jul 7, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

imback82 commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

imback82 commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

imback82 commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

imback82 commented Jul 8, 2020

Uh oh!

SparkQA commented Jul 8, 2020

Uh oh!

imback82 commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 9, 2020

Uh oh!

imback82 commented Jul 9, 2020

Uh oh!

SparkQA commented Jul 10, 2020

Uh oh!

imback82 commented Jul 10, 2020

Uh oh!

SparkQA commented Jul 10, 2020

Uh oh!

imback82 commented Jul 10, 2020

Uh oh!

SparkQA commented Jul 10, 2020

Uh oh!

imback82 commented Jul 17, 2020

Uh oh!

SparkQA commented Jul 17, 2020

Uh oh!

imback82 commented Jul 21, 2020

Uh oh!

gengliangwang commented Jul 24, 2020

Uh oh!

dongjoon-hyun commented Jul 24, 2020

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 24, 2020

Uh oh!

imback82 commented Jul 24, 2020

Uh oh!

edwinalu commented Jul 24, 2020

Uh oh!

SparkQA commented Jul 24, 2020

Uh oh!

SparkQA commented Jul 24, 2020

Uh oh!

wypoon commented Jul 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

imback82 commented Jul 7, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

wypoon commented Jul 25, 2020 •

edited

Loading