[SPARK-34005][CORE] Update peak memory metrics for each Executor on task end #31029
Conversation
|
cc @gengliangwang, @jiangxb1987 and @Ngone51 FYI |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #133662 has finished for PR 31029 at commit
|
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @sarutak .
Merged to master for Apache Spark 3.2.0.
|
@sarutak @dongjoon-hyun shall we port this to branch-3.1 as well? |
|
This is registered as an improvement for 3.2.0, @gengliangwang . |
|
cc @HyukjinKwon since he is the release manager. |
|
Hmm, I registered this improvement for |
|
That was merged a month ago (Dec 20, 2020) before RC cutting. |
|
I am fine with backporting improvements if it's clearly safe. For example, I did backport some changes like c3d8352 too. |
|
+1 for the release manager's decision for this specific commit. |
|
O.K, I'll open a backport PR. |
…ask end ### What changes were proposed in this pull request? This PR makes `AppStatusListener` update the peak memory metrics for each Executor on task end like other peak memory metrics (e.g, stage, executors in a stage). ### Why are the changes needed? When `AppStatusListener#onExecutorMetricsUpdate` is called, peak memory metrics for Executors, stages and executors in a stage are updated but currently, the metrics only for Executors are not updated on task end. ### Does this PR introduce _any_ user-facing change? Yes. Executor peak memory metrics is updated more accurately. ### How was this patch tested? After I run a job with `local-cluster[1,1,1024]` and visited `/api/v1/<appid>/executors`, I confirmed `peakExecutorMemory` metrics is shown for an Executor even though the life time of each job is very short . I also modify the json files for `HistoryServerSuite`. Closes apache#31029 from sarutak/update-executor-metrics-on-taskend. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
… on task end ### What changes were proposed in this pull request? This PR backports SPARK-34005 (#31029). This PR makes `AppStatusListener` update the peak memory metrics for each Executor on task end like other peak memory metrics (e.g, stage, executors in a stage). ### Why are the changes needed? When `AppStatusListener#onExecutorMetricsUpdate` is called, peak memory metrics for Executors, stages and executors in a stage are updated but currently, the metrics only for Executors are not updated on task end. ### Does this PR introduce _any_ user-facing change? Yes. Executor peak memory metrics is updated more accurately. ### How was this patch tested? After I run a job with `local-cluster[1,1,1024]` and visited `/api/v1/<appid>/executors`, I confirmed `peakExecutorMemory` metrics is shown for an Executor even though the life time of each job is very short . I also modify the json files for `HistoryServerSuite`. Closes #31261 from sarutak/SPARK-34005-branch-3.1. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Gengliang Wang <[email protected]>
5 participants
What changes were proposed in this pull request?
This PR makes
AppStatusListenerupdate the peak memory metrics for each Executor on task end like other peak memory metrics (e.g, stage, executors in a stage).Why are the changes needed?
When
AppStatusListener#onExecutorMetricsUpdateis called, peak memory metrics for Executors, stages and executors in a stage are updated but currently, the metrics only for Executors are not updated on task end.Does this PR introduce any user-facing change?
Yes. Executor peak memory metrics is updated more accurately.
How was this patch tested?
After I run a job with
local-cluster[1,1,1024]and visited/api/v1/<appid>/executors, I confirmedpeakExecutorMemorymetrics is shown for an Executor even though the life time of each job is very short .I also modify the json files for
HistoryServerSuite.