Skip to content

[SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation. #27526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LucaCanali wants to merge 5 commits into from
Closed

[SPARK-30775][DOC] Improve the description of executor metrics in the monitoring documentation. #27526

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d39cead
updated following SPARK-30812
LucaCanali Feb 21, 2020
d79b840
Address review comments.
LucaCanali Mar 30, 2020
3ae4a14
typo
LucaCanali Mar 30, 2020
e48afd6
Minor additional corrections to typos and use of case.
LucaCanali Mar 30, 2020
0fc3f2a
Fixed use of case.
LucaCanali Mar 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 49 additions & 5 deletions docs/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -657,16 +657,60 @@ A list of the available metrics, with a short description:

### Executor Metrics

Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information.
Executor metric values and their measured peak values per executor are exposed via the REST API at the end point `/applications/[app-id]/executors`.
In addition, aggregated per-stage peak values of the executor metrics are written to the event log if `spark.eventLog.logStageExecutorMetrics` is true.
Executor metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library.
A list of the available metrics, with a short description:
Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe
the performance metrics of Executor itself like JVM heap memory, GC information.
Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON and Prometheus format.
The JSON end point is exposed at: `/applications/[app-id]/executors`, the Prometheus end point at: `/metrics/executors/prometheus`.
The Prometheus end point is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`).
In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if
`spark.eventLog.logStageExecutorMetrics` is true.
Executor memory metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library.
A list of the available executor metrics, with a short description:

<table class="table">
<tr><th>Executor Level Metric name</th>
<th>Short description</th>
</tr>
<tr>
<td>rddBlocks</td>
<td>RDD blocks in the block manager of this executor.</td>
</tr>
<tr>
<td>memoryUsed</td>
<td>Storage memory used by this executor.</td>
</tr>
<tr>
<td>diskUsed</td>
<td>Disk space used for RDD storage by this executor.</tr>
<tr>
<td>totalCores</td>
<td>Number of cores available in this executor.</td>
</tr>
<tr>
<td>maxTasks</td>
<td>Maximum number of tasks that can run concurrently in this executor.</td>
</tr>
<tr>
<td>activeTasks</td>
<td>Number of tasks currently executing.</td>
</tr>
<tr>
<td>failedTasks</td>
<td>Number of tasks that have failed in this executor.</td>
</tr>
<tr>
<td>completedTasks</td>
<td>Number of tasks that have completed in this executor.</td>
</tr>
<tr>
<td>totalTasks</td>
<td>Total number of tasks (running, failed and completed) in this executor.</td>
</tr>
<tr>
<td>totalDuration</td>
<td>Elapsed time the JVM spent executing tasks in this executor.
The value is expressed in milliseconds.</td>
</tr>
<tr>
<td>totalGCTime</td>
<td>Elapsed time the JVM spent in garbage collection summed in this Executor.
Expand Down