From e9613257194fa39f06ecc9d820716ba43890d772 Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Wed, 5 Aug 2020 18:39:00 +0900 Subject: [PATCH 1/2] Fixed the link to metrics.dropwizard.io. --- docs/monitoring.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/monitoring.md b/docs/monitoring.md index 247957d087fa..842d6b0829e7 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -758,7 +758,7 @@ The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Pr The Prometheus endpoint is experimental and conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`). In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if `spark.eventLog.logStageExecutorMetrics` is true. -Executor memory metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. +Executor memory metrics are also exposed via the Spark metrics system based on the [Dropwizard metrics library](http://metrics.dropwizard.io/4.1.1). A list of the available metrics, with a short description: @@ -962,7 +962,7 @@ keep the paths consistent in both modes. # Metrics Spark has a configurable metrics system based on the -[Dropwizard Metrics Library](http://metrics.dropwizard.io/). +[Dropwizard Metrics Library](http://metrics.dropwizard.io/4.1.1). This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. The metrics are generated by sources embedded in the Spark code base. They provide instrumentation for specific activities and Spark components. @@ -1056,7 +1056,7 @@ activates the JVM source: ## List of available metrics providers Metrics used by Spark are of multiple types: gauge, counter, histogram, meter and timer, -see [Dropwizard library documentation for details](https://metrics.dropwizard.io/3.1.0/getting-started/). +see [Dropwizard library documentation for details](https://metrics.dropwizard.io/4.1.1/getting-started.html). The following list of components and metrics reports the name and some details about the available metrics, grouped per component instance and source namespace. The most common time of metrics used in Spark instrumentation are gauges and counters. @@ -1282,7 +1282,7 @@ Notes: configuration parameter:`spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource` - This source is available for driver and executor instances and is also available for other instances. - This source provides information on JVM metrics using the - [Dropwizard/Codahale Metric Sets for JVM instrumentation](https://metrics.dropwizard.io/3.1.0/manual/jvm/) + [Dropwizard/Codahale Metric Sets for JVM instrumentation](https://metrics.dropwizard.io/4.1.1/manual/jvm.html) and in particular the metric sets BufferPoolMetricSet, GarbageCollectorMetricSet and MemoryUsageGaugeSet. ### Component instance = applicationMaster From ddbf11f5f4073ed378dd51654c1b085afb00128e Mon Sep 17 00:00:00 2001 From: Kousuke Saruta Date: Fri, 14 Aug 2020 00:39:09 +0900 Subject: [PATCH 2/2] Add comment in pom.xml. --- pom.xml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pom.xml b/pom.xml index 6b72e7d13643..9a9c774efbb9 100644 --- a/pom.xml +++ b/pom.xml @@ -145,6 +145,10 @@ 0.9.5 2.4.0 2.0.8 + 4.1.1 1.8.2 hadoop2